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FOREWORD 


This  report  was  prepared  by  the  Systems  Development  Corporation  (SDC)t 
2  500  Colorado  Ave.  ,  Santa  Monica,  California  under  contract  AF19(628) -5166, 
for  the  Director  of  Computers,  Electronic  Systems  Division  (ESD),  as  part  of 
a  continuing  research  effor.t  directed  toward  the  development  of  guidelines, 
standards,  and  techniques  for  improved  management  in  the  field  of  computer 
programming.  This  research  is  a  continuation  of  the  work  that  has  been 
sponsored  by  ESD  since  March  1964.  SDC  report  number  TM-2712/000/00 
applies.  The  Air  Force  Program  Monitor  is  Capt  John  W.  O'Grady,  ESRCC. 

This  report  is  intended  as  a  companion  document  to  the  forthcoming 
"Equations  for  Estimating  the  Cost  of  Computer  Program  Production".  The 
follow-on  document  (due  4  April  1966)  is  intended  for  the  user  whereas  the 
present  work  stresses  the  details  of  the  project  analysis  work  to  date. 

The  research  reported  herein  was  conducted  at  SDC  as  a  part  of  the  Pro¬ 
gramming  Management  Project.  Victor  LaBolle  has  been  the  leader  and  main 
motivating  force  behind  this  work  since  its  inception  in  late  1962,  and  particular 
recognition  is  due  his  contributions.  The  initial  identification  of  factors  that 
affect  the  cost  of  corrputer  programming  was  done  by  L.  Farr  and  B.  Nanus. 

L.  Farr  and  H.  J.  Zagorski  were  responsible  for  the  first  statistical  analysis 
of  historical  cost  data  in  this  project.  N.  E.  Willmorth  also  contributed  to 
these  studies.  The  results  of  this  early  work  are  referenced  in  the 
Bibliography. 

Victor  LaBolle  contributed  material  to  Sections  III,  VIII  and  XX.  Two 
relatively  new  Project  members  prepared  other  portions  of  this  document: 
the  section  dealing  with  the  sensitivity  of  costs  to  changes  in  the  indices  was 
prepared  by  Edward  A.  Nelson;  Thomas  Fleishman  bore  the  main  responsibility 
for  the  Appendices. 

Drs.  Lee  Christie,  Harry  Harman,  Robert  McCormack,  and  John  Walsh 
consulted  with  us  on  special  problems  involving  statistical  analysis. 

The  following  were  among  our  reviewers,  who  bear  no  responsibility  for 
any  deficiencies  of  this  document,  but  whose  contributions  were  appreciated: 

Dr.  Lee  Christie,  Dr.  Harry  H.  Harman,  Dr.  Robert  McCormack,  Jules 
Schwartz,  Clarence  Starkey  and  Dr.  John  Walsh. 

Last,  but  certainly  not  least,  were  the  contributions  of  our  editor,  Ann 
Walker,  and  typists,  Carol  Castillo  and  Diane  Mitchell. 

This  technical  report  has  been  reviewed  and  is  approved. 


CHARLES  A.  LAUSTRUP,  Colonel,  USAF 
Director  of  Computers 

Deputy  for  Engineering  &  Technology  ii 


ABSTRACT 


The  Report  embodies  the  latest  results  of  a  continuing  research 
effort  directed  toward  the  development  of  management  guidelines, 
standards,  and  techniques  in  the  field  of  computer  programming. 

The  work  is  based  upon  earlier  studies  at  System  Development 
Corporation  that  included:  the  definition  of  variables  affecting 
computer  programming  costs;  the  design  of  a  questionnaire  as  an 
aid  to  collecting  data  on  completed  jobs;  and  the  exploratory 
statistical  analysis  of  27  completed  computer  programming  jobs 
to  develop  preliminary  cost-estimation  relationships.  The  present 
Report  is  focused  upon  the  statistical  analysis  of  7k  completed 
computer  programming  jobs  in  terms  of  their  resource-costs  and 
related  variables,  e.g.,  man  months,  computer  hours.  The  primary 
results  developed  in  this  analysis  are:  indices  of  job  difficulty, 
job  type,  development  environment,  and  job  uniqueness;  a 
"costliness"  factor  that  permits  programming  tasks  to  be  ranked  in 
this  respect;  weighted  composites  of  the  indices  for  estimating 
the  cost  of  particular  programming  jobs;  and  scoring  and 
confidence-band  techniques  for  blending  intuitive  managerial 
judgments  with  the  formal  cost- estimation  procedures .  Supple¬ 
mentary  findings  include  indications  of  the  relative  sensitivity 
of  job  cost  to  changes  in  the  values  for  the  indices,  and 
preliminary  comparisons  of  resource  usage  between  programs  produced 
in  machine -oriented  or  procedure -oriented  languages.  Also, 
recommendations  are  made  for  future  research,  including:  the 
collection  of  more  accurate  and  current  data  on  programming  jobs 
during  the  production  cycle,  and  the  development  of  a  census  of 
computer  programming,  to  enable  the  design  of  precise  sampling 
experiments  for  subsequent  analyses. 
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SECTION  I 


SUMMARY  OF  THE  REPORT 


This  Report  presents  the  latest  results  of  a  continuing  research  program 
directed  toward  the  development  of  improved  methods  of  estimating,  collecting, 
and  controlling  costs  for  managers  of  computer  programming  activities.  This 
work  has  been  sponsored  since  March  1964  by  the  Electronic  Systems  Division, 

Air  Force  Systems  Command,  and  is  part  of  the  Programming  Management  Project, 
which  was  established  at  System  Development  Corporation  in  1962.  The  Project 
seeks  to  provide  management  guides,  standards,  and  techniques  in  the  field  of 
computer  programming. 

The  chief  results  of  the  work  reported  here  are  equations  to  estimate  the 
re source- costs  of  computer  programming,  i.e.,  man  months,  elapsed  time,  computer 
hours,  and  number  of  new  instructions.  The  work  to  derive  these  equations 
follows  the  pattern  of  an  earlier  study  and  uses  statistical  analysis  of  experi¬ 
ence  data  obtained  from  completed  programming  projects.  Specifically,  over  the 
past  year,  an  improved  version  of  the  questionnaire  that  was  developed  for 
the  previous  analysis  was  used  to  survey  74  completed  programming  projects, 
once  again  from  System  Development  Corporation.  Of  the  74,  23  were  carried 
over  from  the  earlier  study,  after  additional  information  was  obtained  to 
insure  equivalence  of  the  data.  The  data  base  now  represents  such  diverse 
applications  as  command  and  control,  utility  systems,  management  accounting, 
compilers,  and  research  programs,  and  ranges  from  efforts  of  1  to  over  1600 
man  months. 

As  before,  well-established  techniques  such  as  correlation  analysis  were  used 
to  eliminate  variables  with  spurious  values  and  those  that  showed  little 
predictive  potential.  Multivariate  regression  procedures  were  again  applied 
to  obtain  estimation  equations  for  the  same  resource-costs  that  were  considered 
previously  in  an  earlier  analysis,  i.e.,  man  months,  computer  hours,  new 
instructions,  and  months  elapsed.  The  data  upon  which  the  analysis  was  based 
reflected  the  program  design,  code,  and  test  phases  of  the  production  process. 
System  design,  program  installation,  and  maintenance  were  not  considered. 

To  provide  techniques  that  will  be  useful  to  managers,  the  analysis  was 
intended  to  select  predictor  variables  that  were  known  or  could  be  estimated 
with  reasonable  reliability  early  in  the  program  design  phase,  e.g.,  the 
number  of  subprograms.  Eleven  such  predictors  were  defined  and  grouped  into 
four  indices  that  tend  to  characterize  the  programming  task:  the  Uniqueness 
Index,  the  Job  Difficulty  Index,  the  Development  Environment  Index,  and  the 
Job  Type  Index.  These  indices  were  used  as  predictors  in  multivariate 
regression  analysis  to  obtain  estimation  equations  for  various  resource-costs: 
man  months,  computer  hours,  new  instructions,  and  months  elapsed.  In  addition, 
factor  analysis  procedures  were  used  to  define  an  aggregate  of  resource-cost 
variables  that  can  be  used  to  compare  and  rank  programming  jobs  as  to  their 
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"costliness,"  and  an  estimation  equation  using  the  indices  as  predictors 
was  obtained. 

We  expect  these  indices  to  remain  relatively  constant  from  one  analysis  to 
another  in  a  conceptual  sense,  although  their  components  and  weighting  may 
change.  Moreover,  the  use  of  indices  provides  a  new  and  intuitively  appealing 
way  to  compete  and  rank  programming  tasks  as  to  their  uniqueness,  difficulty, 
development  environment,  and  type. 

A  novel  application  of  the  Stanine  scoring  technique  to  the  construction  of 
confidence  bauds  was  also  used  to  facilitate  the  blending  of  managers 1 
intuitive  and  experiential  judgements  with  the  estimates  computed  from  the 
regression  equations. 

We  regard  the  development  of  task  indices,  their  use  as  predictors  in 
estimating  equations,  and  the  Stanine  scoring  techniques  as  the  major  findings 
of  this  analysis,  and  as  the  elements  of  a  practical  management  tool  for 
estimating  and  comparing  computer  programming  resource- costs .  At  the  same 
time,  we  consider  these  results  transitional  in  the  sense  that  they  are 
based  on  only  programming  projects  from  one  organization.  We  are  now 
taking  steps  to  mitigate  this  constraint  by  collecting  a  variety  of  new  data 
from  a  number  of  government  and  industrial  organizations  for  analysis  in  late 
1965  and  early  1966.  In  addition,  plans  are  being  made  to  design  a  prototype 
cost  collection  system  for  experimental  implementation  in  1966  to  provide  more 
accurate  and  reliable  data  for  analysis  than  is  possible  with  questionnaire 
methods. 

In  addition  to  the  development  work  leading  to  the  major  findings,  the  Report 
also  includes  a  discussion  of  two  supplementary  investigations.  In  one  of 
these,  the  task  indices  were  analyzed  to  determine  their  relative  impact  on 
three  resource-costs:  man  months,  computer  hours,  and  months  elapsed.  This 
study  of  cost  sensitivity  is  a  forerunner  of  future  work  aimed  at  providing 
managers  with  guides  to  making  planning  decisions  with  respect  to  prospective 
programming  jobs.  The  other  supplementary  investigation  concerned  the  effects 
on  program  production  resource-costs  of  machine-oriented  languages  (MOLs)  and 
procedure-oriented  languages  (POLs).  The  comparisons  were  made  in  terms  of 
three  ratios,  Production  Rate  (net  instructions/total  man  months),  Computer 
Usage  Rate  (total  hours/net  instructions),  and  Documentation  Rate  (total  pages/ 
net  instructions),  and  tended  to  confirm  scane  prevalent  opinions  that 
procedure -oriented  languages  are  less  expensive  in  resources.  (These  results 
are  preliminary,  however,  since  only  l4  POL  programs  were  considered.) 

In  summary,  while  substantial  refinements  are  anticipated  with  additional  and 
more  representative  data,  we  believe  that  the  second  analysis  of  data  from 
System  Development  Corporation,  as  presented  in  this  Report,  provides  the 
conceptual  foundation  for  the  estimation  of  resource-costs  by  computer 
programming  managers .  As  in  our  previous  work,  we  recommend  that  the  regression 
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equations  and  Stanine  plots  be  used  as  checks  and  guides  against  estimations 

made  by  other  means,  and  ve  solicit  constructive  feedback  as  to  their 

effectiveness.  We  do  not,  however,  consider  the  estimating  procedures 

described  in  this  Report  as  sufficiently  precise  and  reliable  to  be  used  as 

the  sole  basis  for  management  decisions  with  respect  to  computer  programming. 
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SECTION  II 


AN  INTRODUCTION  TO  THIS  REPORT 


This  Report  is  based  on  research  that  was  performed  as  a  part  of  the  SDC 
Programming  Management  Project,  under  the  sponsorship  of  the  Electronic 
Systems  Division,  Air  Force  Systems  Command^-,  and  represents  a  continuation 
of  the  work  begun  in  March  1964,  and  published  in  the  TM- 144-7  series  of 
documents . 

The  Report  records  the  latest  results  of  a  continuing  effort  to  apply  the 
rational  techniques  of  management  science  to  one  portion  of  the  information 
processing  field— the  process  whereby  a  computer  program  is  designed,  coded, 
and  tested — and,  in  so  doing,  begins  to  augment  recollection  with  data,  and 
intuition  with  analysis.  The  approach  used  to  date  involves  the  collection 
of  detailed  information  on  completed  computer  programming  projects  by 
questionnaire,  and  the  analysis  of  the  results  using  well-established 
statistical  techniques. 

The  work  is  aimed  primarily  at  managers  of  computer  programming  activities, 
who  are  faced  with  the  problems  of  predicting,  measuring,  and  controlling 
economic  (and  technical)  performance  and  value.  While  we  do  not  expect 
that  al  1  managers  will  benefit  equally,  many  will  find  the  relations  that 
are  developed  between  various  aspects  of  computer  programming  to  be  useful. 
Other  managers,  whose  experience  and  intuition  as  yet  exceed  our  results,  may 
find  quantified  support  for  the  relationships  they  have  long  felt  to  exist. 

It  is  clear,  however,  that  the  techniques  developed  herein  will  be  meaningful 
only  to  the  extent  that  they  are  validated  in  practice.  Toward  this  end, 
we  encourage  the  use  of  this  Report  as  a  guide  to  management  decision-making 
and  solicit  constructive  comments  from  any  source.  Moreover,  we  will  gladly 
forward  on  request  the  questionnaires  and  supporting  material  by  which  a 
manager's  estimates  of  a  programming  task,  and  detailed  information  about  the 
actual  production  process,  can  be  collected  and  added  to  our  data  base,  to  be 
reflected  in  later  analysis. 


^Other  past  and  current  research  by  the  Programming  Management  Project  has 
been  supported  by  the  Advanced  Research  Projects  Agency  (ARPA),  the  Office 
of  Naval  Research  (ONR),  the  SHAPE  Technical  Centre  (STC),  and  System 
Development  Corporation.  To  the  extent  that  these  parallel  efforts  in  the 
general  area  of  programming  management  contributed  to  the  ESD- sponsored  work 
reported  in  this  document,  the  organizations  cited  deserve  recognition. 
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To  facilitate  the  user's  retrieval  of  information  we  have  grouped  the  Sections 
into  the  following  parts: 

(1)  A  Summary  of  This  Report 

(2)  An  Introduction  to  This  Report 

(3)  The  Purpose  of  This  Research 

(4)  An  Overview  of  This  Analysis 

(5)  Directions  for  Future  Research 

(6)  The  Technical  Procedures  for  This  Analysis 

(7)  The  Major  Findings  of  This  Analysis 

(8)  Application  of  the  Major  Findings  to  Cost  Estimation 

(9)  Supplementary  Findings 

(10)  Appendices 

(11)  References  and  Bibliography 

The  reader  is  encouraged  to  turn  directly  to  those  portions  of  the  Report  that 
are  of  immediate  interest.  While  the  entire  document  presents  a  comprehensive 
view  of  the  research,  each  part  has  been  written  to  be  interpretable  separately, 
insofar  as  possible: 

a.  Readers  who  are  mainly  interested  in  a  broad  perspective  of  the  work 
of  the  Programming  Management  Project  are  referred  to  parts  (2)  through 

(5),  above. 

b.  Readers  who  are  concerned  with  the  analytical  techniques  used  will  find 
these  discussed  in  part  (6),  above. 

c.  Readers  who  are  primarily  interested  in  the  major  findings  of  this 
Report,  and  their  application,  are  referred  to  parts  (7)  and  (8),  above. 
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SECTION  III 


THE  MANAGEMENT  OF  COMPUTES  PROGRAMMING: 
THE  STATE-OF-THE-ART  SUMMARIZED 


While  the  public  image  of  a  manager  has  undergone  a  substantial  transformation 
in  recent  decades,  there  has  never  been  much  doubt  that  the  ultimate  standard 
by  which  most  managers  are  judged,  and  by  which  they  judge  themselves,  is  the 
economic  performance  of  the  activities  for  which  they  are  responsible.  It 
is  the  ability  to  motivate,  predict,  measure,  and  control  economic  performance 
that  underlies  the  functions  that  traditionally  have  been  given  the  manager 
to  discharge. 

The  immediate  problem  facing  managers  of  information  processing  activities 
today  is  that  reasonably  precise  and  generally  accepted  standards  and  techniques 
by  which  the  economic  performance  of  computers  and  their  applications  can  be 
measured  are  nearly  nonexistent.  Electronic  data  processing  is  new  and 
different  enough  that  most  traditional  management-accounting  guidelines  and 
measures  are  not  very  meaningful.  Decisions  are  made  every  day,  but  they  are 
too  often  based  on  an  ad  hoc  blend  of  experience,  intuition,  and  temerity. 

While  these  qualities  are  certainly  essential  to  some  degree,  they  can  be 
considered  adequate  only  in  the  absence  of  well-founded  methods. 

The  information  processing  manager’s  dilemma  is  further  compounded  by  the 
frenzied  pace  of  computer  technical  development  and  application,  which  shows 
no  signs  of  slackening.  Although  their  projections  vary,  experts  in  the 
field  are  in  complete  agreement  that  the  production  and  installation  of 
computer  hardware  and  the  associated  programs  have  become  very  big  business, 
indeed,  and  seem  destined  to  become  larger  still.  Estimates  from  various 
industry  and  government  sources  indicate  that  national  expenditures  for 
computer  programming  alone  will  range  from  $3  to  $7  billion  annually  by  1970* (l) 

It  is  only  natural  that  the  prospect  of  expending  such  enormous  sums  for 
computer  systems  has  increasingly  focused  attention  on  the  managerial  questions 
of  performance,  quality,  and  effectiveness.  This  is  especially  the  case 
with  the  Federal  Government,  which  is  by  far  the  largest  single  user  of 
electronic  data  processing  equipment  and  services.  Several  significant  studies 
along  these  lines  have  been  issued  recently  by  the  Comptroller  General,  the 
Bureau  of  the  Budget,  and  Committees  of  the  House  and  Senate. (2) 

It  is  generally  recognized,  however,  that  these  reports  represent  more  of 
a  first  step  and  identification  of  problem  areas  for  future  work  than 
definitive  solutions.  A  great  deal  needs  to  be  done  before  the  present 
profusion  of  information  processing  terminology  and  techniques  have  been 
analyzed  and  measures  and  standards  synthesized  that  are  useful  to  managers  in 
the  discharge  of  their  day-to-day  responsibilities. 
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The  difference  between  the  state-of-the-art  in  computer  programming  management- - 
which  is  the  principal  interest  of  the  research  presented  in  this  Report — and 
that  which  prevails  in  more  mature  industries  can  be  perceived  most  clearly 
by  contrasting  the  perspective  of  a  potential  buyer  of  a  computer  program 
with  that  of,  say,  an  automobile. 

First,  an  automobile  is  a  tangible  piece  of  equipment  whose  qualitative 
attributes  can  be  assessed  rather  easily.  A  computer  program,  on  the  other 
hand,  is  delivered  as  a  set  of  documents,  caxds,  tapes,  and  listings,  all 
representing  an  operational  entity  that  cannot  be  seen,  heard,  smelled,  or 
kicked. 

Furthermore,  the  potential  buyer  of  an  automobile  can  make  use  of  the  evalu¬ 
ations  of  popular  journals  in  the  field,  descriptive  literature  from  the 
manufacturers,  or  subjective  reactions  of  friends  who  "own  one."  Most  of 
these  facts  and  opinions  will  be  phrased  in  terms  of  measures  common  to 
automobiles,  such  as  horsepower,  turning  radius,  comfort,  style,  safety, 
optional  accessories,  freedom  from  repair,  and  so  on. 

One  who  is  considering  the  purchase  of  a  computer  program  has  a  much  more 
difficult  task  in  establishing  any  reasonable  criteria  on  which  to  make 
comparisons.  Few  programs  are  ready-made;  most  are  tailored  to  the  needs 
of  the  user.  There  are  almost  no  standards  for  comparing  the  characteristics 
of  programs  with  tlicir  expected  performance.  Usually,  the  design  of  a  computer 
program  is  based  upon  a  description  of  the  job  to  be  done,  but  most  often  the 
characteristics  are  highly  qualitative  rather  than  quantitative. 

Now,  suppose  that  these  contrasts  between  products  are  seen  through  the  eyes 
of  a  production  manager  rather  than  a  potential  buyer.  The  manager  in  the 
automobile  factory  benefits  immediately  from  the  nomenclature  and  procedures 
that  are  standard  throughout  the  automotive  industry.  There  are  techniques 
available  to  predict  and  measure  the  performance  of  the  product,  on  both  a 
unit-by-unit  and  a  sampling  basis;  standards  against  which  the  output  of  both 
men  and  machines  can  be  gauged;  and  an  abundance  of  historical  material  in 
terms  of  which  the  efficiency  of  alternatives  can  be  estimated. 

Because  most  computer  programming  projects  are  thought  of  as  "one  shot"  events, 
the  manager  of  such  an  activity  is  faced  with  an  almost  total  lack  of  standard 
measures  for  the  performance  or  quality  of  products  or  tasks,  or  predictive 
techniques  for  estimating  the  costs  and  manpower  that  will  be  required.  As  a 
manager  tries  to  meet  estimated  budgets  or  schedules  for  a  programming 
job,  however  arrived  at,  the  most  common  ’balancing"  factor  is  quality. 

Since  there  is  no  recognized  and  reliable  yardstick,  quality  can  be  compromised 
in  favor  of  costs  or  schedules.  In  addition,  he  generally  has  no  body  of 
historical  data  on  which  to  rely  in  selecting  alternative  courses  of  action. 
Clearly,  the  computer  programming  manager  would  benefit  immensely  if  the  kinds 
of  management  measures,  standards,  and  techniques  that  are  common  to  many  other 
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industries,  such  as  the  automotive,  could  also  he  developed  for  the  field 
of  computer  programming. 

The  evolution  of  such  managerial  aids  for  computer  programming  has  been 
hindered  by  several  obstacles: 

a.  The  most  apparent  is  simply  the  youth  and  technical  turbulence  of  the 
computer  programming  field.  So  much  has  come  about  so  rapidly  that  compara¬ 
tively  little  time  has  been  spent  on  synthesis  or  management  research. 

b.  There  is  a  general  lack  of  agreement  on  the  concepts  and  terminology 
in  use  throughout  the  computer  programming  field.  While  there  are  many 
glossaries,  such  as  those  prepared  by  the  Association  for  Computing  Machinery 
(ACM),  the  Bureau  of  the  Budget,  and  The  American  Standards  Association,  as 
veil  as  the  forthcoming  IFIP/lCC  Vocabulary  of  Information  Processing, (3) 
they  are  not,  as  yet,  generally  relied  upon,  and  most  organizations  evolve 
their  own  set  of  working  definitions. 

c.  Little  attention  has  been  given  to  the  definition  of  attributes  that 
characterize  the  nature  or  quality  of  a  computer  program  ^ as  a  product.  For 
example,  programmers  use  such  terms  as  "maintainability,1  tightness  of 
coding,"  and  "flexibility,"  but  there  seem  to  be  no  widely  accepted  criteria 
by  which  similar  programs  could  be  compared  in  terms  of  these  factors. 

d.  Present  cost-collection  criteria  seem  to  be  designed  primarily  for 
legal- accounting  purposes •  For  this  reason,  the  historical  data  that  remain 
after  a  programming  project  has  been  completed  are  not  readily  amenable  to 
analysis  in  terms  of  the  broader  needs  of  managerial  planning  and  control. 
Generally  speaking,  except  for  certain  limited  types  of  cost  data,  historical 
information  is  not  kept  in  any  organized  and  cohesive  fashion  at  all. 

What  records  remain  are  not  usually  comparable  from  one  location  to  another 
and,  often,  not  even  within  different  portions  of  the  same  organization. 

e.  Many  of  the  design,  schedule,  and  resource  constraints  that  impinge 
upon  the  computer  programming  process  are  not  well  enough  understood  to  be 
susceptible  to  meaningful  quantification.  The  interrelationships  between  the 
steps  in  the  program  production  process  itself  are  not  well  defined,  and  no 
general  agreement  exists  on  how  individual  variations  should  be  combined  to 
provide  conclusions  that  are  meaningful  in  terms  of  the  program  end-product. 
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SECTION  IV 


RESEARCH  INTO  THE  MANAGEMENT  OF  COMPUTER  PROGRAMMING: 
MAJOR  PROJECT  ACHIEVEMENTS  AND  AIMS 


In  summary,  there  is  an  urgent  need  for  ways  of  making  what  in  current 
parlance  are  called  "cost-effectiveness"  decisions  with  regard  to  computer 
programming  specifically  (and  information  processing  generally).  But  the 
ability  to  measure  cost  effectiveness  presupposes  reasonably  precise  and 
generally  accepted  measures  of  cost  and  measures  of  effectiveness,  which,  in 
turn,  presupposes  a  pool  of  structured  and  comparable  historical  data  from 
which  such  measures  can  be  derived  rationally;  this,  finally,  presupposes 
the  commonly-held  notions  about  concepts,  processes,  and  terminology  that 
would  allow  such  a  data  base  to  be  collected.  As  we  have  observed  previously, 
none  of  these  prerequisites  have  been  more  than  minimally  satisfied. 

For  more  than  three  years,  the  Programming  Management  Project  has  been  working 
at  each  of  the  three  links  in  the  research  chain  leading  to  measures  of  cost- 
effectiveness  for  computer  programming: 

a.  At  the  most  elementary  level,  we  have  modeled  the  development  of 

computer  programs  as  a  process,  and  dissected  it  to  define  a  common  sequence 
of  tasks.  (The  most  recent  result  of  this  effort  has  been  a  detailed  Planning 
Guide  for  Computer  Program  Development (  4 ) . )  — — — 

b.  We  have  developed  and  refined  a  detailed  questionnaire,  and  used  it 
to  collect  a  substantial  body  of  historical  data,  both  quantitative  and 
qualitative,  from  a  wide  range  of  completed  programming  projects.  (Although 
the  data  were  originally  confined  to  SDC  experience,  questionnaires  are  now 
being  completed  by  a  variety  of  Air  Force  and  industrial  organizations  that 
do  significant  amounts  of  computer  programming. ) 

c.  We  have  applied  well-established  statistical  techniques  to  the 
resulting  data  base.  These  have  led  to  the  selection  of  cost  factors  that 
seem  especially  significant,  and  that  form  the  basis  for  experimental  cost 
collection  systems. 

d.  We  have  developed  equations  for  estimating  computer  programming  costs 
through  the  design,  code,  and  test  phases.  These  equations,  and  various  other 
methods  of  comparing  observations  with  experience  are  available  to  managers 
for  use  on  an  experimental  basis. 

e.  We  have  engaged  several  of  the  major  accounting/consulting  firms  to 
survey  the  state-of-the-art  of  computer  programming  management  in  government 
and  industry,  and  to  suggest  techniques  to  synthesize  the  research  results 
with  accounting  practice. (  5 ) 
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f.  We  have  done  some  preliminary  work  toward  the  development  of  management 
standards  in  terms  of  which  the  performance  and  quality  of  programming  products 
can  be  measured. 

We  see  these  efforts  as  a  long-term  attempt  to  evolve  products  of  two  kinds: 
first,  a  generally  accepted  and  applicable  way  of  collecting  resource-costs  on 
computer  programming  projects,  for  managerial  instead  of  legal  record-keeping 
purposes;  second,  standards,  guidelines,  and  techniques  that  will  permit  the 
manager  to  use  the  statistical  distillation  of  past  experience  to  predict  and 
control  the  performance  of  computer  programming  activities  within  his 
jurisdiction. 

By  the  end  of  Fiscal  Year  1967?  we  hope  to  have  progressed  to  the  point  where 
results  of  these  two  types  can  be  made  available  on  an  "operational"  basis  for 
the  first  time.  However,  it  is  doubtful  that  these  research  products  will  be 
definitive  or  final  by  the  end  of  Fiscal  Year  1967,  or  even  later.  There  is 
far  too  much  innovation  taking  place  in  the  computer  programming  field,  e.g., 
procedure-oriented  languages,  time-sharing,  etc.,  to  allow  much  hope  along 
these  lines. 

On  the  other  hand,  the  progress  that  has  been  made  in  other  fields  does  suggest 
that,  while  managerial  problems  are  not  usually  susceptible  to  precise  solution 
within  specified  periods  of  time,  substantial  improvements  can  be  made  if  a 
rational  approach  is  combined  with  practical  experience  in  a  step-by-step  way. 

Our  practical  object  in  this  work  is,  quite  simply,  to  help  the  computer 
programming  manager  do  a  better  job--better  tomorrow  than  yesterday,  and 
better  in  1966  than  in  1965*  At  the  present  state-of-the-art,  it  is  only 
natural  that  no  single  technique  or  result  will  serve  all  purposes  or  solve 
al  1  problems.  A  variety  of  approaches,  some  more  elegant,  some  less  so, 
will  need  to  be  developed  along  the  way.  But  the  primary  criterion  of  our 
work  is  usefulness,  and  of  that,  managers  of  computer  programming  must  be 
the  final  judges. 

The  growing  awareness  of  the  need  for  improved  management  standards  and 
techniques  in  computer  programming  (and  information  processing  generally) 
has  also  stimulated  a  variety  of  work  in  these  areas.  In  this  context,  we 
view  our  research  as  one  of  a  community  of  related  projects,  all  focused 
upon  one  or  more  EDP  management  problems.  Some  of  the  more  notable  efforts 
involve : 


a.  The  National  Bureau  of  Standards,  which  has  recently  formed  a  Computer 
Science  and  Technology  Center.  The  plans  for  the  Center  include  work  to  guide 
government  agencies  in  the  development,  measurement,  and  test  of  standards  for 
equipment  and  techniques,  including  programming  languages. 
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b.  The  office  of  the  Director  of  Defense,  Research  and  Engineering 
(DDR&E),  Department  of  Defense,  in  which  a  number  of  individuals,  especially 
Dr.  James  Ward,  have  long  been  active  in  EDP  management  problems. 

c.  The  Electronic  Systems  Division  (ESD),  of  the  Air  Force  Systems 
Command  (AFSC),  which  has  sponsored  a  variety  of  exploratory  studies  in  this 
area  by  Auerbach  Corporation,  Planning  Research  Corporation,  and  others. (6  ) 

d.  The  MITRE  Corporation  and  the  System  Development  Corporation,  under 
sponsorship  of  Electronic  Systems  Division,  have  been  engaged  in  a  program  to 
delineate  configuration  management  procedures  for  computer  programming. ( 7  ) 

e.  The  Air  Force  Systems  Command  has  been  developing  a  Cost  Information 
System  (CIS),  as  a  part  of  its  Cost  Management  Improvement  Program.  The  aim 
is  to  facilitate  the  collection  of  comparable  and  maximally  meaningful  cost 
data  for  all  projects,  including  computer  programming,  under  AFSC  sponsorship. 
McKinsey  and  Company  provided  consulting  support  to  this  work. 

f.  Various  other  individuals  and  organizations,  such  as  Brandon  and 
Associates,  the  Diebold  Group,  the  Standards  and  Procedures  Association,  the 
Data  Processing  Management  Association,  and  the  major  auditing/consulting 
firms,  have  also  been  active  in  this  field. 
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SECTION  V 


THE  OBJECT  AND  APPROACH  OF  THIS  ANALYSIS 


The  first  exploratory  statistical  analysis  of  historical  cost  data  from 
completed  computer  programming  projects  was  undertaken  in  1964-5(8),  and  had 
several  aims: 

a.  To  define  the  major  variables  that  affect  the  cost  of  computer 
programming. 

b.  Using  a  questionnaire  based  on  these  variables,  to  collect  quantitative 
and  qualitative  data  for  a  sample  of  completed  computer  programming  projects. 

c.  To  demonstrate  the  usefulness  of  certain  statistical  techniques, 
especially  multivariate  regression,  as  a  means  of  exploring  the  relationships 
between  potential  predictor  variables  that  describe  the  programming  job, 

(e.g.,  the  development  environment,  the  staff,  etc.),  and  project  resource- 
costs  (e.g.,  man  months,  computer  hours,  etc.). 

A  total  of  27  completed  programming  projects,  all  from  the  System  Development 
Corporation,  were  analyzed,  based  on  questionnaire-acquired  data.  Considering 
the  almost  complete  lack  of  comparable  historical  information  on  computer 
programming  available  when  the  work  began,  the  research  along  these  lines 
was  quite  successful: 

a.  Over  a  hundred  variables  affecting  computer  programming  costs  were 
identified. 

b.  A  comprehensive  data- collection  questionnaire  was  developed,  and 
refined  by  application  in  the  field. 

c.  A  small  but  diverse  and  relatively  unique  body  of  information  on 
programming  efforts  was  gathered  and  analyzed. 

d.  Linear  regression  equations  were  derived  to  estimate  some  of  the  major 
cost  parameters,  e.g.,  man  months,  computer  hours,  etc. 

On  the  other  hand,  the  derivation  of  meaningful  results  from  such  a  small, 
experimental  sample  required  that  some  rather  severe  compromises  be  made 
during  the  analysis:2 


2 

The  comments  that  follow  are  mainly  our  self-criticism  of  the  work,  but 
also  generally  reflect  the  constructive  feedback  from  our  readers.  The 
emDhasis  is  on  the  analysis  of  the  first  SDC  data  collection. 
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a.  Since  effective  regression  analysis  generally  requires  substantially 
more  data  points  than  variables  (the  opposite  of  the  situation  that  initially 
existed),  about  two- thirds  of  the  original  100  variables  had  to  be  eliminated — 
some  on  the  basis  of  statistical  relationships,  some  intuitively — before 

formal  analysis  began.  However,  the  winnowing  process  left  much  to  be  desired  as 
to  technique. 

b.  The  remaining  independent  variables  were  not  easy  to  group  as  measures 
of  the  program  production  task,  e.g.,  the  interrelation  between  individual 
predictors  such  as  the  size  of  the  data  base  and  the  number  of  document  types, 
on  the  one  hand,  and  computer  hours  on  the  other,  as  an  indication  of  total 
program  cost,  was  not  apparent. 

c.  Again,  since  so  few  data  points  were  available,  no  thought  could  be 
given  to  subdividing  the  population  by  types  of  programs.  All  the  data 
points  were  thrown  into  a  common  pool  and  analyzed  together. 

d.  The  small  number  of  data  points  also  eliminated  any  possibility  of 
exploring  nonlinear  regression  relationships  directly,  at  least  in  a  statistical 
way.  Effective  nonlinear  regression  analysis  generally  requires  an  even  higher 
proportion  of  data  points  to  variables  than  does  linear  analysis. 

e.  The  fact  that  the  few  available  data  points  tended  to  be  clustered 
at  the  ends  of  a  rather  wide  range  of  values,  and  that  exponential  relation¬ 
ships  between  many  of  the  cost  and  predictor  variables  appeared  to  be  present, 
required  the  elimination  of  several  extreme  outliers,  and  the  transformation 
of  several  variables  into  logarithmic  form. 

f .  Finally,  the  indecisiveness  of  many  of  the  statistical  relationships 
concerning  the  predictor  variables  required  the  use  of  "Number  of  New 
Instructions,"  itself  an  unknown,  as  a  predictor.  While  the  number  of 
instructions  can  perhaps  be  estimated  more  easily  in  advance  of  program 
production  than  can  man  months,  computer  hours,  etc.,  the  approach  taken  was 
not  wholly  satisfactory  from  a  practical  point  of  view. 

The  objectives  of  the  present  analysis  were,  quite  simply,  to  go  beyond  the 
first  effort  in  several  ways: 

a.  By  using  an  improved  questionnaire  to  collect  data. 

b.  By  gathering  a  larger  base  of  SDC  data,  more  amenable  to  statistical 
evaluation. 

c.  By  developing  techniques  that  would  tend  to  overcome  the  shortcomings 
of  the  first  analysis,  e.g.,  meaningful  ways  of  aggregating  predictor  variables. 

These  aims  were  broadened  further  when  the  granting  of  an  Air  Force  Systems 
Command  Report  Approval  Number  provided  the  authority  to  collect  data  from  a 
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variety  of  military  and  industrial  organizations( 9  )  for  analysis  in  late 
1965  and  early  1966.  As  a  result,  the  second  analysis  of  SDC  data  also 
became  a  test  vehicle  for  many  of  the  concepts  and  techniques  that  were 
developed  with  the  military  and  industrial  data  bases  in  mind. 

From  more  than  100  SDC  programming  projects  initially  under  consideration 
(including  the  27  used  in  the  first  analysis),  a  total  of  74  were  included  in 
the  final  data  base  for  this  analysis. 3  These  represented  a  variety  of 
programming  tasks  including  management  data  processing,  command  and  control, 
research  compilers  and  utility  routines,  information  retrieval,  etc. 

Although  the  74- point  data  base  was  nearly  three  times  as  large  as  that 
available  for  the  preceding  analysis,  the  total  number  of  points  was  still 
considered  too  small  for  a  major  effort  to  be  expended  on  a  study  of  sub¬ 
populations.  Instead,  every  effort  was  made  to  retain  all  valid  data  points 
in  the  analysis  and  thus  to  develop  relationships  that  seemed  to  be  meaningful 
for  many  types  of  computer  programming,  and  the  techniques  that  would  be 
applicable  for  the  study  of  later,  larger  data  collections.  For  example: 

a.  Only  14  of  the  74  data  points  were  coded  in  a  procedure-oriented 
language  (POL),  a  sample  too  small  and  diverse  to  permit  predictive  relation¬ 
ships  to  be  inferred  with  any  substantial  reliability .  Rather  than  eliminate 
the  POL-coded  points,  we  used  the  number  of  machine- language  instructions 
after  compilation  as  a  variable  that  seemed  reasonably  comparable  to  the 
number  of  machine- language  instructions  that  resulted  when  the  other  60  points 
were  coded  in  a  M0L\  and  analyzed  all  74  points  on  this  basis.  The  price 
paid  for  this  compromise,  of  course,  was  the  increase  in  variation  that  the 
POL  points  contributed  to  the  total  data  base,  which  made  estimation  more 
difficult . 

b.  For  most  of  the  major  variables,  e.g.,  man  months,  computer  hours, 
words  in  data  base ,  etc . ,  the  few  large  data  points  were  greatly  outnumbered 


^Projects  were  eliminated  from  further  consideration  for  a  variety  of  reasons, 
including  dates  for  availability  of  the  data,  low  reliability  of  the  responses, 
etc . 


We  realize  that  the  number  of  machine- language  instructions  after  compilation 
would  have  been  somewhat  less  if  the  programs  in  question  had  been  coded  in 
machine -oriented  language  directly.  Some  experts  say  that,  in  the  case  of 
JOVIAL,  this  increase  in  "tightness"  varies  from  10  to  15  percent.  Once 
again,  we  chose  to  accept  the  increase  in  variance  rather  than  introduce 
further  uncertainties  at  this  time.  Evaluations  of  the  effects  of  POLs 
on  program  production  will  be  considered  in  future  studies. 
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by  the  relatively  smaller  programming  projects.  To  increase  the  cohesion 
of  the  data  base,  we  transformed  most  of  these  variables  logarithmically 
rather  than  eliminate  the  outliers. 

Within  the  major  objectives  of  deriving  useful  relationships  from  a  single, 
representative  data  base,  the  major  technical  emphasis  for  this  analysis 
was  placed  on  the  grouping  of  significant  variables  in  ways  that  seemed 
intuitively  and  statistically  meaningful.  However,  the  derivation  of  resource 
cost  predictors  for  computer  programming  is  necessarily  a  long,  iterative 
analytical  process,  which  involves  many  changes  in  the  variables  that  are 
considered  and  their  relative  weighting.  Sequences  of  many  variables,  which 
change  from  one  analysis  to  another,  may  not  be  intuitively  attractive  to 
the  user. 

On  the  other  hand,  although  information  on  many  variables  is  collected,  most 
reflect,  perhaps,  a  half  dozen  major  aspects  of  the  programming  job:  the 
difficulty  of  the  design  and  coding,  the  development  environment,  the  staff, 
the  equipment,  and  so  on.  It  is  much  more  meaningful  to  think  in  terms  of 
indices  of  these  various  aspects  of  the  programming  that  will  tend  to  remain 
reasonably  constant  from  one  analysis  to  another,  although  their  composition 
and  weighting  may  change.  Such  indices  will  also  allow  ranking  or  scoring  of 
programming  tasks  as  to  difficulty,  type,  etc.,  based  on  demonstrably 
meaningful  composites  of  significant  variables. 

We  have  developed  four  such  task  indices  in  this  analysis,  and  have  used  them 
to  estimate  computer  programming  resource-costs.  Another  grouping  of  cost 
variables  was  also  obtained  to  allow  the  comparison  of  programming  efforts  as 
to  their  "costliness."  We  believe  that  these  aggregates  of  variables,  which 
are  discussed  in  detail  in  Sections  XV  and  XVI, are  a  more  practical  tool  for 
management  use  than  was  available  previously,  and  consider  the  estimating 
equations  presented  in  Section  XVII  as  the  major  finding  of  this  Report. 
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SECTION  VI 


THE  TECHNICAL  PLAN  FOR  THIS  ANALYSIS 


1.  Background.  It  is  a  fact  of  statistical  analysis  that  an  abundance  of 
quality  data  allows  the  researcher  comparative  freedom  as  to  techniques.  On 
the  other  hand,  the  most  refined  methods  can  hardly  compensate  for  a  dispro¬ 
portion  of  variables  to  data  points,  as  was  the  case  in  the  preceding  study 
of  this  series.  Although  a  total  of  Jk  data  points  is  not  sufficient  for  all 
analytical  purposes,  it  does  cross  the  threshold  into  the  realm  of  meaningful 
inferences  about  the  computer  programming  process.  The  technical  plan  that 
was  developed  for  this  analysis  reflects  both  the  larger,  more  representative 
base  of  data  and  the  broader,  more  ambitious  objectives  outlined  previously. 

To  clarify  the  presentation,  the  research  results  are  discussed  separately,  in 
Sections  IX  through  XX. 

It  should  be  noted  that  the  Technical  Plan  is  simply  a  guide  to  insure  that 
the  analyses  of  different  batches  of  data  are  comparable ,  and  that  no  fruitful 
steps  are  neglected.  The  analytical  process  may  dictate  the  skipping  of  one 
step  or  another,  or  the  iteration  through  a  certain  procedure  several  times. 

In  this  sense,  the  Technical  Plan  is  simply  an  analytical  framework,  a  check¬ 
list,  in  terms  of  which  the  evaluation  of  the  data  can  be  conducted. 

2.  Outline  of  the  Technical  Plan.  In  general  terms,  the  development  of  a 
cost  prediction  model  for  a  single  population  of  computer  programming  projects' 
the  main  object  of  this  analysis--was  carried  out  in  accordance  with  the 
following  sequence: 5 

a.  Data  Quality  Control,  Correction  and  Formatting: 

(1)  Validation,  error  correction,  etc. 

(2)  Analysis  of  outliers,  modification  of  data. 

b.  Initial  Definition  of  Dependent  and  Independent  Variables 

c .  Formal  Analysis: 


^The  technical  plans  for  the  other,  smaller  studies  that  were  conducted  in 
parallel  with  the  main  analysis — e.g.,  development  of  resource-cost  ratios, 
and  MOL  versus  POL  comparisons- -are  presented  together  with  the  particular 
results,  in  Sections  XIX  and  XX.  The  technical  plan  for  the  study 
of  subpopulations  will  be  developed  in  subsequent  reports  dealing  with  these 
matters . 
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(1)  Scatterplot  analysis  of  independent  versus  dependent  variables. 

(2)  Correlation  analysis  of  all  variables. 

(3)  Factor  analysis  of  independent  and  dependent  variables. 

(4)  Regression  analysis  of  dependent  factor  measures. 

(5)  Intuitive  modification  of  factor  measures. 

(6)  Development  of  independent  variable  aggregates  (indices). 

(7)  Regression  analysis  of  independent  variable  aggregates  (indices) 
against  dependent  variables. 

(8)  Regression  analysis  of  dependent  variable s  and  factors  against 
independent  variables,  factors,  and  indices. 

d.  Analysis  of  Regression  Model-Fitting  Errors: 

(1)  Exclusion  of  undesirable  data  points. 

(2)  Application  of  curve-fitting  techniques  other  than  the  least- 
squares  method. 

(3)  Search  for  overlooked  variables,  factors,  or  indices. 

3.  Discussion  of  the  Technical  Plan.  Although  a  detailed  review  of  the 
theory  underlying  each  step  of  the  Technical  Plan  is  beyond  the  scope  of  this 
Report,  and  can  be  found  in  the  Bibliography,  a  brief  exposition  follows: 

a.  Data  Quality  Control,  Correction,  and  Formatting: 

(1)  Validation,  error  correction,  etc:  The  particular  statistical 
techniques  used--factor  and  multivariate  regression  analysis--are  especially 
sensitive  to  spurious  data,  e.g.,  from  misinterpretation  of  the  questionnaire, 
transcription  errors ,  etc . 

Prior  to  the  beginning  of  formal  analysis,  all  data  are  checked  for  accuracy 
and  quality. 

(2)  Analysis  of  outliers,  modification  of  data:  Ideally,  the  data 
for  each  variable  should  be  continuously  and  normally  distributed,  and  the 
data  points  should  be  independent  of  one  another.  Since  these  conditions  are 
seldom  met,  a  certain  departure  from  the  ideal  can  be  tolerated.  The  closer 
to  the  ideal,  however,  the  more  trustworthy  will  be  the  statistical  results. 
(Although  variables  with  discontinuities  and  very  skewed  distributions  are 
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avoided  whenever  possible,  they  may  be  included,  on  occasion,  to  permit  early 
state-of-the-art  results.)  The  biasing  effects  of  pronounced  skewness  can 
usually  be  mitigated  by  transformation,  e.g.,  to  logarithmic  values. 

Apart  from  poor  distributions,  some  variables  may  include  data  that  are 
somewhat  outlandish  with  respect  to  the  median.  Since  such  data  have  a 
disproportionately  degrading  effect  on  factor  and  multivariate  regression 
analyses,  which  deal  with  weighted  values,  it  is  common  practice  to  apply 
various  statistical  measures  to  determine  whether  or  not  the  outliers  are 
part  of  the  same  population.  A  variety  of  rules  are  available  to  facilitate 
this  determination,  e.g.,  those  by  Dixon,  Grubbs,  Murphy,  Thompson,  and 
Tukey ( 10 ) •  Particular  extreme  data  are  usually  replaced  by  a  value  closer  to 
the  median  of  the  variable,  i.e.,  Winsorized,  to  avoid  an  undue  prejudice  of 
the  subsequent  analysis.  In  some  cases,  if  many  of  the  values  for  the  variables 
that  describe  a  single  programming  effort  are  extreme,  the  data  point  may  be 
deleted. 


b.  Initial  Definition  of  Dependent  and  Independent  Variables.  The 
variables  are  separated  into  two  logically  distinct  classes:  those  to  be 
predicted,  and  those  which  are  to  serve  as  predictors.  When  historical  data 
are  being  analyzed,  care  must  be  exercised  to  insure  that  predictor  variables 
will  be  available  in  measurable  form  or  can  be  estimated  reliably,  when  the 
prediction  equations  are  to  be  used. 

4.  Formal  Analysis: 

a.  Scatterplot  Analysis  of  Independent  and  Proposed  Dependent  Variables. 
Visual  analysis  of  the  relationship  between  pairs  of  independent  and  proposed 
dependent  variables  often  reveals  special  relationships,  such  as  nonlinearities 
or  potentially  spurious  outliers,  that  are  difficult  to  determine  otherwise . (ll ) 

b.  Correlation  Analysis  of  All  Variables.  Correlation  analysis  involves 
the  study  of  the  intercorrelations  between  all  the  variables,  i.e.,  the 
correlation  matrix.  Although  factor  and/or  multi variate  regression  techniques 
are  more  powerful  and  are  relied  upon  for  the  main  analysis,  the  correlation 
values  themselves  provide  insights  that  could  be  overlooked  otherwise . (ll ) 

Two  types  of  relationships  are  especially  important:  first,  that  the  inter¬ 
correlations  between  the  independent  variables  are  as  small  as  possible,  i.e., 
that  the  predictors  are  maximally  independent;  second,  that  the  intercorrelations 
between  each  independent  and  the  dependent  variables  are  as  large  as  possible, 
i.e.,  that  the  so-called  "validity"  of  each  independent  variable  be  maximal 
with  respect  to  the  dependent  variables. (11 ) 

c.  Factor  Analysis  of  Independent  and  Dependent  Variables.  Technically, 
factor  analysis  is  a  method  for  studying  the  intercorrelations  among  a  group 
of  variables  to  define  the  common  influences  that  are  present.  The  specific 
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aim  is  to  reduce  the  dimensionality  of  the  intercorrelation  matrix  from  one 
composed  of  variable-versus-variable  correlations  to  a  smaller,  more  easily 
interpretable  one  composed  of  factor- versus- variable  correlations.  The 
principal  components  model  first  derived  by  Hotelling(l2 )  and  the  varimax 
factor  rotation  procedure  developed  by  Kaiser(l3 ) ,  are  well-established 
techniques  for  dealing  with  a  problem  of  this  type. 

d.  Regression  Analysis  of  Independent  Factor  Measures.  To  obtain  factor 
measures,  the  multivariate  regression  process  is  applied  to  secure  the  best 
combination  of  variables  for  defining  each  factor.  In  this  process,  the 
factors  become  weighted  composites  of  the  component  variables. (l4 )  (Since 
the  factors  have  no  known  mean  and  standard  deviation,  arbitrary  values  are 
assigned,  e.g.,  in  this  Report,  a  mean  of  100  and  a  standard  deviation  of 

20  have  been  found  to  be  convenient . ) 

e.  Intuitive  Modification  of  Factor  Measures.  In  using  factor  analysis, 
there  is  no  guarantee  that  the  combinations  of  variables  derived  will  be 
intuitively  meaningful  or  interpretable.  It  is  always  possible  and  sometimes 
desirable  to  modify  factor  analysis  results  in  the  light  of  practical 
considerations.  The  value  of  engaging  in  this  sort  of  adjustment  depends, 

of  course,  on  the  ingenuity  of  the  analyst.  When  there  are  strong  practical 
considerations  for  making  a  modification,  however,  the  change  should  be  made. 

f.  Development  of  Independent  Variable  Aggregates  (indices).  It  is 
possible  to  group  independent  variables  solely  on  the  basis  of  intuition,  and 
to  weed  out  those  whose  intercorrelations  with  other  predictors  in  the  group 
are  higher  than  their  correlation  with  cost.  The  result  is  the  opposite  of 

a  factor:  a  combination  of  highly  Dredictive  variables  that  express  intuitively 
similar  but  statistically  different  oarts  of  the  variation,  rather  than 
reinforcing  the  same  underlying  contribution. 

g.  Regression  Analysis  of  Independent  Variable  Aggregates  (indices) 

Against  the  Dependent  Variables.  As  in  step  "4d,"  regression  analysis  is 
applied  to  obtain  the  weighted  composites  of  each  index  that  are  most  effective 
as  predictors. 

h.  Regression  Analysis  of  Dependent  Variables  and  Factors  Against 
Independent  Variables,  Factors,  and  Indices.  Once  the  dependent  and  independent 
variables  have  been  defined,  the  primary  aim  of  multivariate  regression 
analysis  is  to  provide  a  rational  basis  for  the  selection  and  weighting  of 

the  predictors .  (l4  ) 

A  nonlinear  regression  approach  may  be  applied,  depending  on  the  nature  and 
amount  of  data  and  the  relative  advantage  of  this  technique. 
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i.  Analysis  of  Regression  Model- Fitting  Errors 


(1)  Exclusion  of  undesirable  points:  A  desirable  follow-up  to  the 
definition  of  a  prediction  equation  is  to  evaluate  the  nature  of  the  variance 
that  is  not  accounted  for  by  the  model.  Outlier  points  can  often  be  identified 
using  bivariate  plots  of  actual  versus  predicted  values.  The  elimination  of 
such  points,  when  justified  on  the  grounds  that  the  outliers  are  truly  part 

of  another  population,  will  permit  an  improvement  in  the  confidence  of  the 
predictions,  based  on  a  reanalysis  of  the  data  base. 

(2)  Application  of  curve-fitting  techniques  other  than  the  least- 
squares  method:  Although  the  least-squares  method  of  fitting  the  prediction 
line  is  the  most  common,  other  approaches,  e.g.,  equalizing  the  variance  of 
the  observed  and  predicted  values  around  the  estimation  line,  may  lead  to  a 
distribution  of  variations  that  is  more  consistent  over  the  values  of  interest. 
Often,  the  loss  in  predictive  power  and  confidence  is  small  enough  to  permit 
an  adjustment  of  this  type  to  be  made. 

(3)  Search  for  overlooked  variables,  factors,  or  indices:  Occasionally, 
a  cluster  of  data  points  may  lie  significantly  away  from  the  prediction  line. 

This  usually  suggests  that  an  entire  factor,  index,  or  variable,  that 
represents  a  characteristic  common  to  all  the  points,  has  been  overlooked  in 

the  definition  of  the  model. 

5.  Application  of  the  Technical  Plan.  In  general,  the  analysis  was  carried 
out  according  to  the  plan.  Certain  steps,  however,  were  left  for  subsequent 
analyses .  These  are : 

a.  A  more  intensive  application  of  factor  analysis  methodology  to  define 
additional  aggregates  was  not  carried  out.  This  will  probably  be  done  when 
the  data  from  government  and  industry  have  augmented  the  7^-point  base. 

b.  Nonlinear  regression  analysis  was  not  applied,  primarily  because  the 
number  of  data  points  was  insufficient.  Also,  the  transformation  of  many 
variables  to  logarithmic  form  produced  relationships  that  were  essentially 
linear.  Nonlinear  regression  analysis  may  be  used  with  larger  data  bases,  if 
it  seems  advantageous. 

c.  The  analysis  of  regression  model- fitting  errors,  step  "4i,"  was  not 
done,  primarily  due  to  a  lack  of  sufficient  time.  While  the  intent  of  this 
analysis  was  to  consider  all  74  data  points  together,  the  deletion  of  one  or 
two  programs  with  extreme  values  could  improve  estimation  confidence  for  the 
remainder.  (This  is  not  the  same  as  step  "2a,"  which  deals  with  outliers  ^ 
for  individual  variables.)  We  intend  to  repeat  this  analysis  after  step  "4i" 
has  been  done,  in  a  subsequent  study. 


21 

(Page  22  Blank) 


SECTION  VII 


AN  EVALUATION  OF  THE  PRESENT  RESEARCH 


As  indicated  previously,  the  intent  of  this  analysis  was  to  progress  beyond 
the  earlier  work  and,  hopefully,  to  achieve  more  meaningful  and  reliable 
results.  As  will  be  substantiated  in  later  sections  dealing  with  the 
findings,  these  objectives  have  been  realized  to  a  considerable  degree. 

At  the  same  time,  there  is  no  denying  that,  against  the  whole  spectrum  of 
computer  programming  activities  with  which  the  manager  must  deal,  much 
remains  to  be  done. 

For  instance,  the  research  to  date  has  not  encompassed  the  so-called  system 
design,  system  test,  and  maintenance  phases,  which  fall  at  either  end  of  the 
program  design,  code,  and  assembly  test  cycle.  Neither  has  the  cost  of 
managerial  and  support  overhead  beyond  the  first  level  of  programmer  super¬ 
vision  been  included.  While  these  omissions  can  be  attributed  to  the 
difficulty  of  defining  generally  applicable  and  comparable  ways  of  measuring 
these  resource-costs,  there  is  no  question  that  a  wholly  realistic  picture 
of  the  computer  programming  process  cannot  be  portrayed  in  their  absence. 
(Some  managers  are  correct  in  observing  that,  in  certain  cases,  e.g.,  very 
large,  complex  program  systems  with  many  changes,  the  phases  we  have  omitted 
may  be  more  expensive  than  those  that  have  been  included. ) 

A  related  problem  is  that  of  estimating  costs  before  the  program  design 
phase,  e.g.,  before  the  start  of  system  design.  While  reliable  cost 
estimation  is  difficult  at  any  step  in  the  programming  process,  the  earlier 
the  predictions  are  made,  the  more  hazardous  the  task. 

Another  deficiency  is  the  lack  of  an  effective  way  of  measuring  the  relative 
capability  of  the  programming  staff.  While  various  combinations  of  average 
experience  and  positional  rating  have  been  tried  in  the  last  analysis  and 
this  one,  an  effective  yardstick  in  this  critical  area  remains  to  be 
developed. (15)  The  organization  of  the  programming  staff,  e.g.,  project 
or  "assembly-line"  approach,  has  also  not  been  considered. 

Also,  there  is  the  matter  of  machine  and  language  comparability.  No 
meaningful  and  reliable  way  has  been  found  to  gauge  quantitatively  the 
effects  that  different  computers  and/or  compilers  have  on  equivalent 
programming  jobs.  These  considerations  are  being  complicated  further  by 
the  proliferation  of  new  equipment  and  new  techniques,  such  as  language, 
time-sharing,  etc.  Although  their  adequacy  and  compatibility  in  terms  of 
our  research  needs  is  not  clear  at  the  present  time,  some  experimental 
ranking  systems,  e.g.,  Auerbach's  vector  approach(l6),  may  be  useful  in 
this  respect. 
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For  these  particular  shortcomings,  we  can  express  at  least  the  hope  of 
alleviation  -through  additional  research;  but  there  is  the  far  more  funda¬ 
mental  and  disturbing  criticism  that  the  present  data  base,  for  all  its 
diversity  and  uniqueness,  still  represents  a  collection  of  information  that 
is  almost  exclusively  after-the-fact.  Questionnaires  were  filled  out  only 
after  a  programming  project  was  completed  and,  what  is  more,  included  many 
queries  on  which  data  were  not  normally  collected  by  the  respondents  at 
all,  and  which  could  therefore  only  be  answered  in  a  "best-recollection" 
fashion. 

While  these  after-the-fact  data  have  been  quite  useful  in  allowing  vis  to  get 
a  handhold  on  the  relationships  that  affect  computer  programming  activities, 
the  data  are  clearly  not  adequate  for  the  development  of  the  reasonably 
precise  techniques  that  are  required  for  management  control.  The  accuracy 
of  responses  in  several  cases  such  as  Number  of  Instructions  Discarded  Due 
to  Errors,  is  difficult  to  determine.  Moreover,  the  data  do  not  provide 
much  help  in  analyzing  the  variation  of  resource-costs  through  the  program 
production  process,  i.e.,  the  flow  of  expenses  from  one  phase  to  another. 

To  go  the  next  step,  to  define  reliable  relationships  between  the  various 
cost  parameters  and  translate  them  into  techniques  that  will  be  useful  to 
managers  of  computer  programming  projects  in  the  performance  of  their  day- 
to-day  responsibilities,  we  know  that  a  substantially  more  reliable  and 
continuous  data  base  is  required.  Such  a  data  base  can  only  be  gathered 
within  existing  cost  collection  systems,  on  a  regular  basis,  during  the  life 
of  a  programming  project.  We  call  this  "on-line  cost  collection." 

The  accumulation  of  costs  during  program  development  rather  than  after-the- 
fact,  however  desirable,  cannot  become  a  reality  until  such  a  procedure  is 
not  only  meaningful  technically  but  feasible  economically.  The  collection 
of  information  itself  has  a  cost;  and  the  notion  of  increasing  the  frequency 
and  comprehensiveness  of  computer  program  cost  data,  whatever  its  analytical 
potential,  is  simply  not  practical  until  certain  prerequisites  have  been 
satisfied: 

a.  The  number  of  variables  on  which  data  are  to  be  collected  must  have 
been  reduced  to  a  minimum; °  and  evidence  as  to  the  effectiveness  of  these 
variables  in  predicting  costs  must  have  been  demonstrated  from  both  the 
scientific  and  managerial  points  of  view. 


Of  course,  this  "minimum"  will  change  as  the  programming  field  changes,  and 
new  relationships  are  developed  and  demonstrated. 
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b.  The  variables  themselves,  and  the  methods  of  recording  and  collecting 
them,  must  have  been  adequately  defined  and  made  compatible  with  the  account¬ 
ing  systems  and  techniques  that  are  generally  prevalent,  or  their  logical 
extensions. 

c.  Since  sampling  of  computer  programming  projects  is  an  economic 
necessity  (it  is  not  practical  to  collect  extensive  data  from  all  programming 
efforts),  there  must  be  reasonable  evidence,  from  a  statistical  viewpoint, 
that  the  analysis  of  the  resulting  data  will  yield  inferences  that  are 
meaningful  in  the  broader  population  of  programming  tasks. 

The  impact  that  these  requirements  have  on  the  prospects  for  future  research 
in  this  area  is  the  focus  of  the  following  Section. 
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SECTION  VIII 


SOME  AREAS  FOR  FUTURE  STUDY 


The  approach  taken  in  this  work  is  based  on  the  proposition  that  the  analysis 
of  comparable  data  on  the  resource-costs  of  computer  programming  projects--so 
long  as  it  is  done  on  a  reasonably  economical  basis — is  central  to  the  long¬ 
term  development  of  meaningful  management  standards  and  techniques  in  this 
field. 

In  the  previous  Section,  we  have  commented  on  the  shortcomings  of  the  ways  in 
which  data  were  collected  for  this  analysis,  e.g.,  after-the-fact  information 
leaves  something  to  be  desired  as  to  reliability  and  validity.  The  "on-line" 
approach  promises  substantially  to  alleviate  this  deficiency,  and  to  provide 
the  method  whereby  a  more  comprehensive  assortment  of  information  can  be 
collected  in  parallel  with  the  program  production  process. 

In  addition,  our  present  plans  will  carry  us  beyond  the  work  in  this  Report 
in  several  ways: 

a.  The  data  that  are  being  collected  from  government  and  industry  will 
result  in  a  substantially  larger  and  more  representative  data  base,  and  will 
likely  permit  the  analysis  of  separate  subpopulations,  i.e.,  types  of 
programming  jobs. 

b.  The  results  of  the  analyses  of  questionnaire-acquired  data  will  be 
blended  with  accounting  practice  and  terminology  into  a  prototype  cost  model 
for  computer  programming,  which  will  be  experimentally  applied  to  the  on-line 
data  collection. 

A  more  fundamental  problem  remains,  however.  The  mechanics  of  data  collection 
and  analysis  notwithstanding,  what  types  of  computer  programming  tasks,  in 
what  proportion,  should  be  considered  for  the  on-line  data  phase?  By  what 
criteria  is  the  composition  of  the  analytical  data  base  to  be  determined? 

Practically  speaking,  no  standards  were  applied  in  the  accumulation  of  the 
present  data  base,  apart  from  questionnaire  accuracy  and  completeness. 

Managers  in  various  areas  of  the  System  Development  Corporation  were  asked 
to  supply  representative  samples  of  the  programming  done  under  their  juris¬ 
diction,  but  no  data  points  were  rejected  on  the  grounds  that  they  were 
non-representative . 

As  a  result,  the  data  base  tends  to  resemble  a  cross-section  of  programming 
practice,  e.g.,  there  are  many  smaller  programs  and  few  larger  ones.  Since 
this  research  is  in  its  early  stages,  with  an  object  of  defining  relationships 
common  to  programming  generally,  a  data  base  of  this  type  is  considered 
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acceptable.  As  the  techniques  employed  herein  become  more  familiar  and  tested, 
however,  and  as  the  emphasis  moves  more  toward  results  that  are  useful  to 
computer  programming  managers  in  the  performance  of  their  day-to-day  responsi¬ 
bilities,  some  criteria  must  clearly  be  enforced  as  to  the  composition  of  the 
data  base  from  which  the  end-product  relationships  are  to  be  derived. 

One  approach  is  to  collect  increasing  amounts  of  data  from  organizations  of 
all  types  and  sizes  that  do  computer  programming.  As  the  data  base  becomes 
larger  and  the  sources  of  information  more  diverse,  the  resulting  collection 
should  approximate  the  programming  that  is  being  done  in  the  field.  In  the 
absence  of  any  criteria  other  than  size,  rather  massive  amounts  of  data  would 
have  to  be  collected  to  assure  that  every  type  of  task  was  adequately  repre¬ 
sented-^. g.  ,  machine-oriented  languages,  procedure-oriented  languages,  on¬ 
line  usage,  time-sharing,  jobs  of  different  sizes,  complexities,  applications, 
and  so  on. 

Realistically,  the  time  when  a  representative  data  base  on  computer  programming 
could  effectively  be  gathered  by  the  process  of  "exhausting"  the  population  no 
longer  seems  economically  feasible,  and  too  much  innovation  is  taking  place  in 
the  computer  field  to  suggest  that  data  collection  problems  are  becoming  simpler 
in  this  respect. 

The  obvious  alternative  is  to  rely  on  some  sort  of  sampling  process  whereby 
the  whole  population  of  computer  programs  is  adequately  represented  with  the 
collection  of  a  relatively  small  amount  of  data. 

Of  course,  this  is  not  as  easy  as  it  sounds.  Technically,  a  random  sample 
(which  is  what  we  are  referring  to)  is  one  in  which  every  item  in  the  popula¬ 
tion  has  an  equal  chance  of  being  selected,  and  in  which  the  selection  of  one 
item  or  another  is  independent,  i.e.,  there  is  no  element  of  collusion  in  the 
selection  process. 

For  our  purposes,  the  first  restriction  is  by  far  the  most  serious.  In  the 
previous  analysis  and  in  this  one,  all  the  data  were  gathered  from  different 
parts  of  one  organization,  the  System  Development  Corporation.  We  are  now 
collecting  questionnaire  data  from  about  fifteen  government  and  industry 
sources.  While  this  is  a  step  in  the  right  direction,  we  have  no  reason  to 
assume,  for  instance,  that  all  computer  programming  projects  in  the  population 
have  an  equal  chance  of  selection.  Quite  the  contrary;  we  know  they  do  not. 

The  essential  problem  is  the  absence  of  a  standard  by  which  the  represent¬ 
ativeness  of  the  sample  can  be  judged  and,  therefore,  the  extent  to  which 
statistical  relationships  derived  from  the  sample  can  be  relied  upon.  The 
way  in  which  this  problem  has  been  surmounted  in  the  face  of  similar  difficulties 
has  been  to  rely  on  a  census.  In  political  forecasting  or  marketing,  for 
example,  the  "population"  is  as  complex  and  dynamic  as  that  of  computer  program¬ 
ming.  Very  precise  samples  are  drawn,  and  inferences  made,  by  defining  the 
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proportion  of  different  classes  of  data  that  should  be  included  using  a  census 
of  the  population,  and  then  selecting  at  random  within  each  category  until  the 
correct  ratio  is  achieved. 

Fortunately,  in  political  forecasting  or  marketing,  the  rudimentary  categories 
are  relatively  clear  and  generally  acknowledged:  there  are,  for  instance, 
men,  women,  Caucasians,  Negroes,  Orientals,  employed,  unemployed.  Republicans, 
Democrats,  city-dwellers,  farmers,  and  so  on;  and  the  assumption  that  these 
differences  are  related  to  voting,  or  purchasing,  seems  reasonable.  But  there 
are  no  comparable  and  generally  applicable  or  accepted  categorizations  for 
computer  programming,  (it  is  not  clear  that  many  of  the  prevalent  distinctions, 
e.g.,  real  time,  on-line,  command  and  control,  etc.,  are  sufficiently  precise 
or  meaningful  for  census  purposes . ) 

It  is  our  hope  that,  out  of  the  research  in  this  Project  and  other  parallel 
efforts  in  the  field,  meaningful  and  reasonably  precise  classes  of  computer 
programming  will  be  defined  and  validated  over  a  period  of  time.  Such  devel¬ 
opments,  in  turn,  will  permit  the  essentials  of  a  census  of  computer  programming 
tasks  to  begin  to  be  gathered.  While  the  prospect  for  quick  or  easy  progress 
in  this  area  is  minimal,  we  suggest  that  the  direction  proposed  is  essential 
to  the  future  of  effective  management  research  in  the  field. 

For  the  more  immediate  future,  several  other  areas,  beyond  those  that  are  a 
part  of  our  present  plans,  seem  to  merit  additional  study: (17) 

a.  A  significant  and  largely  unexplained  aspect  of  programming  cost 
estimation  and  control  concerns  the  management  of  program  changes  and  reprogram¬ 
ming.  This  type  of  work  accounts  for  a  large  percentage  of  total  programming 
costs  today,  and  will  increase  as  the  number  of  installations  grow.  Much  of 
the  motivation  for  modifications  or  additions  to  existing  programs  stems  from 
the  frequent  introduction  of  new  computers  and  configurations  (e.g.,  time¬ 
sharing),  languages,  applications,  etc.  Since  both  the  extent  to  which  changes 
will  be  made,  and  nonessentia],  but  useful,  improvements  in  the  program  made  along 
with  the  changes  is  difficult  to  forecast,  the  impact  on  total  costs  of  the 
maintainability  of  the  program  and  the  stability  of  the  design  have  not  been 
given  the  attention  they  deserve.  Since  the  programming  activities  are  essen¬ 
tially  the  same,  one  could  hypothesize  that  the  factors  influencing  the  cost 

of  reprogramming  would  be  the  same  as  those  in  the  present  analysis.  But 
additional  factors  that  reflect  changes  to  the  existing  products,  e.g.,  the 
data  base,  documentation,  program  design,  etc.,  would  have  to  be  considered. 

b.  The  lack  of  quality  or  performance  measures  for  computer  programming 
products  was  also  cited  in  the  previous  study.  Present  criteria  in  this  area 
tend  to  rely  on  such  relatively  tangible  matters  as  whether  or  not  the  functions 
noted  in  the  specifications  are  performed  by  the  program,  whether  the  program 
operates  within  the  existing  time  and  storage  limitations,  and  so  on.  For  the 
most  part,  the  question  of  comparing,  let  alone  predicting,  the  quality  or 
maintainability  of  a  program  product  is  beyond  the  state-of-the-art. 
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c.  The  effect  of  the  development  computer  and  its  configuration  on  the 
cost  and  quality  of  program  production  should  be  thoroughly  investigated.  At 
present,  most  computers  are  used  for  both  ADP  operations  and  program  develop¬ 
ment.  The  number  of  users  and  applications  at  a  given  facility  may  be  quite 
varied,  however;  e.g.,  a  time-shared  installation,  or  restricted,  e.g.,  a 
single-user,  single-application  military  complex.  There  are  several  rather 
obvious  areas  in  which  the  characteristics  of  the  development  computer  could 
lead  to  trade-offs  that  affect  programming  costs.  For  example: 

(1)  Procedure -oriented  versus  machine -oriented  languages. 

(2)  Time-shared  versus  "closed  shop"  operations. 

(3)  Semiautomated  support  versus  additional  computer/utility  systems. 

(4)  Centralization  versus  decentralization  of  computing  facilities. 

d.  The  amount  and  nature  of  the  documentation  that  is  required  to  augment 
the  program  product  has  a  major  impact  on  total  cost.  Considerable  controversy 
prevails  as  to  the  quantities  and  types  of  documents  that  are  preferable. 

Although  some  work  has  been  done  to  systematize  programming  documentation, 
less  effort  has  been  directed  toward  determining  the  cost -versus -value  trade¬ 
offs  of  various  alternatives,  e.g.,  decision  tables,  flow  charts,  key  words, 
prose,  etc. 

e.  Testing  of  computer  programs  is  often  the  most  expensive,  time- 
consuming,  and  costly  part  of  the  production  process.  Little  has  been  done 

to  systematize  the  planning  and  execution  of  program  tests,  and,  most  important, 
to  provide  the  manager  with  guidelines  to  measure  both  cost  and  quality  of  the 
result.  A  related  question  is  the  degree  to  which  quality  control  of  design, 
programming,  documentation,  etc.,  can  reduce  the  effort  that  need  be  devoted 
to  testing. 

f.  The  initial  analysis  and  design  of  the  information  processing  system, 
often  called  "system  design,"  is  a  costly  part  of  development  that  merits 
additional  study.  Since  the  products  of  this  phase  are  the  source  documents 

for  the  remainder  of  the  programming  process,  they  exert  a  considerable  influence 
on  cost  and  quality.  Although  much  theory  on  system  design  exists,  practice 
appears  to  differ  significantly.  There  is  a  need  to  unify  theory  with  practice 
and  provide  comparative  measures  of  quality,  value,  and  cost. 
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SECTION  IX 


PROCEDURES  FOR  THE  COLLECTION  AND  QUALITY  CONTROL  OF  THE  DATA 

1.  The  pata  Collection  Questionnaire.  For  this  analysis, „the  questionnaire 
used  in  the  first  study  (l8)  was  simplified  and  clarified.  The  seven  divisions 
were  retained: 

(1)  Summary  of  Costs 

(2)  Operational  Requirements  and  Design 

(3)  Program  Design  and  Production 

(4)  Data  Processing  Equipment 

(5)  Programming  Personnel 

(6)  Management  Procedures 

(7)  Development  Environment 

However,  a  number  of  significant  changes  were  made  as  a  result  of  the  earlier 
study.  For  instance: 

a.  The  responses  to  several  questions  were  felt  to  be  consistently 
unreliable.  This  led  to  deletion  in  some  cases.  The  "Data  Accuracy  Index," 
for  example,  required  the  respondent  to  estimate  the  accuracy  of  his  informa¬ 
tion.  The  results  indicated  that,  although  the  concept  was  appropriate,  the 
answers  were  not  statistically  useful,  and  the  question  was  deleted. 

b.  Certain  questions  were  amplified  to  provide  more  meaningful  data. 

The  Total  Number  of  Instructions  Discarded,  for  example,  was  separated  accord¬ 
ing  to  the  reasons  for  the  rejection,  e.g.,  programming  errors  or  operational 
changes. 


c.  Questions  that  seemed  intuitively  meaningful  but  subject  to  misinter¬ 
pretation  were  supplemented  with  a  brief  definition  of  the  terms  causing 
difficulty.  In  the  case  of  the  system  complexity  ranking,  for  example,  each 
of  the  five  levels  among  which  a  respondent  could  choose  was  briefly  described. 

d.  Questions  that  revealed  little  or  no  statistical  significance,  or 
proved  to  be  redundant  to  other,  superior  questions,  were  considered  for 
elimination,  e.g.,  the  security  classification  of  the  resultant  documentation. 

e.  The  format  of  the  questionnaire  was  improved,  e.g.,  photographic 
reduction  was  used  for  definitions,  consistency  in  appearance  was  emphasized, 
etc. 
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This  was  done  by  L. 


Farr  and  H.  J.  Zagorski. 
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The  definition  of  a  "data  point,"  and  restriction  of  the  questions  to  the 
program  design,  code,  and  assembly  test  portions  of  the  production  process 
were  retained.  The  rationale  in  these  areas  is  given  in  the  report  of  the 
previous  analysis. ( 19 ) 

One  important  obstacle  to  a  more  sweeping  revision  of  the  questionnaire  was 
the  desire  to  retain,  for  the  second  analysis  of  System  Development  Corporation 
data,  the  27  points  that  had  been  used  previously.  A  supplement  to  the  earlier 
questionnaire  was  designed  to  make  the  first  data  base  comparable,  and  each 
of  the  original  respondents  was  asked  to  provide  additional  information.  In 
3  of  the  27  cases,  sufficiently  reliable  records  were  not  available  and  the 
data  points  were  dropped  from  further  consideration. 

The  questionnaire  was  used  in  the  form  shown  in  Appendix  I.  (The  questions 
that  were  included  in  the  Supplement  to  the  first  questionnaire  are  also 
noted. ) 

2.  Methods  For  Data  Collection.  As  in  the  earlier  data  collection,  no 
sample  design,  in  the  classical  sense,  was  used.  Instead,  managers  through¬ 
out  the  System  Development  Corporation  were  encouraged  to  complete  question¬ 
naires  for  a  representative  assortment  of  the  programming  activities  under 
their  jurisdiction.  The  diversity  of  activities  within  the  Corporation,  and 
the  widespread  response,  greatly  aided  our  efforts  to  obtain  a  well-balanced 
data  base. 

Over  100  programming  jobs  were  considered  for  the  sample  at  one  time  or 
another.  Various  considerations,  e.g.,  tightness  of  schedules,  unavai lability 
of  personnel  to  complete  the  questionnaire,  extension  of  the  test  phase  beyond 
the  sample  cut-off  date,  etc.,  gradually  eliminated  potential  respondents 
until  a  total  of  78  completed  questionnaires — including  23  from  the  first 
study--were  available  for  formal  analysis. 

3.  Quality  Control  of  the  Data  Base.  Over  a  period  of  weeks,  each  response 
for  each  variable  was  evaluated  by  the  Project  staff.  Where  there  was  some 
doubt  as  to  the  appropriateness  of  the  answer,  e.g.,  the  respondent  misunder¬ 
stood  the  question,  wrote  in  the  wrong  numbers  by  mistake,  neglected  to 
provide  an  answer,  etc.,  the  question  was  flagged  for  further  study.  In  each 
such  case,  the  respondent  was  contacted,  if  possible,  and  the  particular 
problems  were  discussed  in  detail.  In  most  cases,  suspicious  answers  were 
resolved  by  this  procedure.  In  some  instances,  seemingly  spurious  data  were 
demonstrated  to  be  valid  and  were  retained. 

Despite  all  efforts  along  these  lines,  certain  responses  could  not  be  justified, 
either  because  no  adequate  records  existed,  or  because  the  only  person  who 
could  have  illuminated  the  situation  was  no  longer  available.  Four  question¬ 
naires  were  dropped  from  further  consideration  because  the  majority  of  their 
responses  fell  into  the  suspicious  category.  Certain  variables,  e.g., 
frequency  of  program  operation,  were  misinterpreted  often  enough  that  they 
were  eliminated  from  the  analysis  for  all  data  points. 
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Finally,  for  8l  of  about  7T00,  or  about  one  percent  of  the  items  in  the  data 
base,  when  most  of  the  responses  for  a  questionnaire  were  considered  valid 
and  all  attempts  at  clarification  of  the  suspicious  or  nonexistent  information 
for  specific  answers  failed,  values  were  estimated  by  the  Project  staff.  In 
each  case,  these  were  intended  to  be  nonbiasing,  i.e.,  average  for  the  type 
of  program  involved.  The  values  that  were  estimated  are  noted  in  Appendix  III. 

The  74  data  points  that  remained  for  the  formal  analysis  represented  a  variety 
of  programming  tasks,  e.g.,  command  and  control,  compilers,  information 
retrieval,  operational  utility  systems,  research  programs,  management  informa¬ 
tion  systems,  and  others. 
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SECTION  X 


THE  CHARACTERISTICS  OF  THE  DATA  BASE 


1.  The  Age  of  the  Data  Base.  An  important  question  regarding  the  data  that 
have  been  collected  is  whether  they  fairly  represent  the  computer  programming 
state -of -the -art,  or  whether  innovations  in  techniques  and  the  equipment 
environment  have  rendered  the  data  obsolete  before  the  analytical  results  can 
be  applied.  While  no  complete  answer  can  be  given,  we  have  made  some  compar¬ 
isons  on  the  very  conservative  assumption  that  the  date  on  which  program 
design  began  is  an  indication  of  the  currency  of  the  data,  (in  fact,  of  course, 
for  many  programming  jobs,  one  or  two  years  elapse  between  the  start  of  program 
design  and  the  completion  of  the  test  phase.)  On  this  basis  the  second  sample 
of  data  from  the  System  Development  Corporation  is  now  significantly  more  up- 
to-date  than  the  first: 


First  SDC  Data  Base  Second  SDC  Data  Base 

Average  Age  at  Present  3*2  years  2.0  years 

By  comparison,  the  average  age  of  the  data  base  that  is  now  being  collected 
from  government  and  industry  is  only  0.8  years(9  )• 

It  is  important  to  note  that  the  average  age  of  the  first  data  base  from  the 
System  Development  Corporation  at  the  time  it  was  analyzed  was  about  two  years, 
the  same  as  the  second  data  base.  After  the  analysis  of  the  government  and 
industry  data  is  completed,  the  average  age  of  the  data  will  have  increased 
to  about  1.5  years,  which  seems  about  the  minimum  lag  between  the  average 
program  design  start  date  and  the  completion  of  analyses  for  the  present 
questionnaire  methods  of  data  collection.  The  on-line  data  collection  tech¬ 
niques  described  in  Section  IV  axe  expected  to  shorten  significantly  this  lag 
between  the  state-of-the-programming-art  and  the  availability  of  the  research 
results . 

2.  Distribution  of  the  Data.  A  careful  study  of  the  data  distribution  for 
each  variable  revealed  that,  except  for  those  coded  in  single  digits  (e.g.,  1, 
2,  or  3),  or  percentiles,  most  were  exponentially  distributed,  i.e.,  a  great 
many  small  values  relatively  close  together  at  the  lower  end  of  the  scale,  and 
few,  rather  large  and  sparsely  distributed  points  at  the  high  end.  The  fact 
that  effective  multivariate  regression  analysis  presupposes  a  reasonably 
continuous  and  normal  distribution  of  data  made  the  mitigation  of  the 
exponentially  scattered  points  a  major  concern. 

One  approach  would  have  been  deliberately  to  collect  enough  data  on  large 
programming  tasks  to  balance  the  data  base.  This  would  have  been  difficult, 
due  to  the  relative  scarcity  of  the  more  complex  and  sizeable  programs,  and 
the  unstructured- sample  approach  was  selected. 


Another  alternative,  which  was  actually  used  in  the  first  analysis,  was  to 
delete  several  of  the  more  extreme  outliers.  However,  as  indicated  in 
Section  V,  we  did  not  consider  the  elimination  of  data  on  such  grounds  to  be 
in  consonance  with  the  main  objectives  of  this  analysis,  i.e.,  to  demonstrate 
relationships  and  analytical  techniques  that  were  common  and  applicable  to  all 
types  of  programming  tasks.  We  realized  that  the  development  of  the  most 
effective  management  devices  required  that  we  reduce  the  total  variation  by 
studying  various  subpopulations  of  the  data  base  separately.  Since  a  much 
larger  and  broader  collection  of  information  was  expected  with  the  addition  of 
data  from  government  and  industry,  and  since  the  74-point  base  was  considered 
to  be  at  the  threshold  of  statistical  reliability,  we  chose  to  postpone  the 
consideration  of  subpopulations  for  the  time  being  and  use  this  analysis  as  a 
test  vehicle  for  demonstrating  the  techniques  that  were  to  be  applied  in  sub¬ 
sequent  efforts.  On  this  basis,  no  individual  data  points  were  eliminated 
from  consideration  due  to  their  extreme  values. 

We  would  also  like  to  emphasize,  however,  that  the  viewpoint  that  is  charac¬ 
teristic  of  this  analysis,  i.e.,  combining  all  the  data  into  a  common  base, 
does  not  preclude  refinement  of  the  results  with  extreme  values  removed.  We 
intend  to  develop  these  relationships  in  subsequent  reports  of  this  series. 
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SECTION  XI 


THE  FORMATION  AND  TRANSFORMATION  OF  VARIABLES 


1.  The  Formation  of  Additional  Variables.  One  of  the  directions  taken  with 
this  analysis  was  the  development  of  ratios  to  indicate  relative,  rather  than 
absolute,  differences,  e.g.,  the  variation  in  computer  hours  expended  for 
comparably  sized  programs.  Although  there  was  not  sufficient  time  in  this 
analysis  to  explore  these  ratios  in  detail,  we  expect  them  to  be  useful  in 
later  studies,  especially  those  of  subpopulations.  (The  ratios  that  were 
constructed  as  a  part  of  this  analysis  are  shown  in  Appendix  II,  as  variables 
109  through  lU8.) 

Three  of  the  ratios,  Production  Rate,  Computer  Usage  Rate,  and  Documentation 
Rate  were  applied  to  the  comparison  of  the  effects  on  program  production  of 
machine-oriented  and  procedure-oriented  languages,  and  are  discussed  in  detail 
in  Section  XX. 

2.  Transformation  of  Variables.  To  compensate  for  the  exponential  distribution 
of  many  variables,  a  logarithmic  transformation  was  applied.  The  effect  can 

be  illustrated  by  Figures  1  and  2,  showing  the  frequency  distribution  of  man- 
months  before  and  after  the  transformation. 

The  logarithmic  transformation  clearly  pulls  the  extreme  values  toward  the 
median  enough  to  provide  a  reasonably  symmetric  unimodal  distribution 
approaching  normality. 

The  effect  of  the  logarithmic  transformation  is  illustrated  further  with  the 
following  frequency  distributions  for  the  variables  Computer  Hours,  Months 
Elapsed,  and  Number  of  New  Instructions.  (Once  again,  for  comparative  purposes, 
the  distributions  are  shown  with  a  mean  of  100  and  a  standard  deviation  of  20.) 

3.  Coding  of  the  Variables.  The  title,  identity  number,  and  coding  of  each 
variable  used  in  this  analysis  is  provided  in  Appendix  II. 
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FIGURE  2 

FREQUENCY  DISTRIBUTION  OF  LOG1Q  TOTAL  MAN  MONTHS 


FIGURE  3 

FREQUENCY  DISTRIBUTION  OF  LOG1Q  COMPUTER  HOURS 
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FIGURE  k 

FREQUENCY  DISTRIBUTION  OF  LOG1Q  MONTHS  ELAPSED 


FIGURE  5 

FREQUENCY  DISTRIBUTION  OF  LOG1Q  NEW  INSTRUCTIONS  (MACHINE) 


Uo 


SECTION  XII 


SCATTERPLOT  AND  CORRELATION  ANALYSIS: 
IDENTIFICATION  OF  DEPENDENT  VARIABLES 


1.  Scatterplot  Analysis.  After  logarithmic  transformations  had  been  applied 
to  the  data  where  necessary,  over  400  computer-prepared  scatterplots  of 
bivariate  relationships  were  prepared  and  evaluated.  The  primary  purpose  was 

to  provide  visual  assurance  that  the  transformed  variables  were  well  distributed, 
e.g.,  that  extreme  outliers  that  could  bias  subsequent  analyses  were  not 
present. 

Several  of  the  more  interesting  plots  are  shown  in  Figures  6,  7>  and  8.  To 
facilitate  interpretation,  the  67-percent  confidence  bands,  in  which  that 
proportion  of  values  would  fall  for  normally  distributed  data,  are  shown. 

Note  the  obvious  linearity  of  the  relationship  for  the  transformed  variables 
(i.e.,  for  raw  values  plotted  on  log-log  paper).  Of  course,  these  plots  are 
for  illustrative  purposes  only;  the  confidence  bands  are  too  wide  to  be  useful 
for  meaningful  estimation. 

2.  Correlation  Analysis.  The  first  correlation  matrices  were  composed  of 
104  rows  and  columns,  which  essentially  represented  the  variables  taken 
directly  from  the  questionnaires,  after  transformations  had  been  accomplished 
as  necessary. 

The  early  eliminations  were  based  on  correlations  that  were  either  relatively 
high  or  low.  Since  the  Standard  Error  of  a  null  correlation  coefficient  for 
74  data  points  is  l//Y4  or  about  0.12,  the  search  for  prediction  variables 
focused  on  those  that  exhibited  validities  of  at  least  0.20  with  likely 
dependent  variables,  e.g.,  man  months.  Potential  predictors  that  showed 
intercorrelations  above  0.20  were  also  evaluated  for  potential  redundancy. 

In  all  these  cases,  correlation  analysis  was  used  as  a  guide  to  highlight 
relationships  that  could  bias  later  analysis.  Since  regression  analysis 
computer  programs  with  a  subsetting  feature  were  available,  these  were  also 
applied  to  compare  the  sequence  in  which  the  variables  were  selected  for 
prediction.  The  final  decisions  as  to  whether  a  variable  should  be  eliminated 
from  further  consideration  were  made  by  the  Project  staff,  based  on  the 
statistical  results  and  the  projected  utility  of  the  particular  variable. 

The  original  104  variables  were  gradually  winnowed,  using  the  procedures 
outlined,  until  69  variables  remained.  These  are  listed  in  Appendix  v,  and 
the  deletions  can  be  determined  by  a  comparison  with  Appendix  II. 
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FIGURE  6 

RELATIONSHIP  BETWEEN  NEW  INSTRUCTIONS  AND  TOTAL  MAN  MONTHS 
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TOTAL  MAN  MONTHS 
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FIGURE  7 

RELATIONSHIP  BETWEEN  TOTAL  COMPUTER  HOURS  AND  TOTAL  MAN  MONTHS 
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TOTAL  COMPUTER  HOURS 


NEW  MACHINE  INSTRUCTIONS 


FIGURE  8 

RELATIONSHIP  BETWEEN  NEW  MACHINE  INSTRUCTIONS  AND  TOTAL  COMPUTER  HOURS 
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3.  Identification  of  Dependent  Variables.  Based  on  the  correlation  analysis 
results  obtained  in  the  previous  steps,  four  primary  dependent  variables  were 
identified  for  the  remainder  of  the  analysis.  These  were  the  same  as  in  the 
previous  study  of  this  series  (TM-IH7/OOI/OO) . 

Variable  Number  Title 

1  Logic  Total  Man  Months 

9  Logic  Total  Computer  Hours 

20  k°e10  ^eW  Instruc"kions  (Machine) 

121  Log10  Months  ElaPsed 

It  is  important  to  note  that  many  other  dependent  variables  could  have  been 
added  to  this  list.  Each  regression  analysis  essentially  represents  a 
separate  study  effort,  however,  and  time  constraints  did  not  permit  considera¬ 
tion  of  additional  dependent  variables. 

Based  on  the  above,  all  other  variables  (among  the  69)  that  were  not  appro¬ 
priate  as  predictors,  e.g.,  tended  to  duplicate  the  selected  dependent 
variables,  were  considered  for  elimination.  The  result  was  a  list  of  40 
"independent"  variables,  shown  in  Appendix  VT.  The  deletions  can  be  identified 
by  a  comparison  with  Appendix  V. 

The  validities  of  the  more  important  predictors  with  the  dependent  variables, 
and  the  Costliness  Factor,  are  provided  in  Appendix  IV. 


SECTION  XIII 


MULTIVARIATE  REGRESSION  USING  INDIVIDUAL  PREDICTORS 


In  what  amounted  to  a  repeat  of  the  earlier  analysis  (TM-l447/00l/00),  the 
4  dependent  cost  variables  were  regressed  against  the  40  independent  predictors, 
and  weighted  composites  of  3  to  8  variables  were  obtained. 

One  very  significant  difference  from  the  previous  analysis  was  that  Number  of 
New  Instructions,  itself  an  unknown,  was  no  longer  being  used  as  an  independent 
variable.  The  predictors,  in  each  case,  either  are  known  or  can  be  estimated 
with  reasonable  reliability  at  the  start  of  program  design.  While  the  multiple 
regression  coefficients  (an  indication  of  the  extent  to  which  the  variance  of 
the  dependent  variable  is  explained  by  the  predictors)  are  not  directly  com¬ 
parable  with  those  of  the  previous  analysis,  due  to  the  absence  of  Number  of 
New  Instructions,  the  fact  that  at  least  one  multiple  regression  coefficient 
between  0.70  and  0.80  was  obtained  for  all  predictions  was  considered  quite 
encouraging. 

For  illustrative  purposes,  several  of  the  equations  derived  in  this  manner  are 
shown  in  Appendix  VIII.  The  others  are  not  included  since  the  techniques  that 
are  recommended  for  prediction  involve  the  use  of  task  indices,  which  are 
developed  in  the  following  Sections. 
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SECTION  XIV 


FACTOR  ANALYSIS: 

DEVELOPMENT  OF  THE  COSTLINESS  FACTOR 


1.  Background.  As  noted  in  Section  IV,  one  of  the  main  deficiencies  in 
the  first  study,  which  we  sought  to  remedy  in  this  analysis,  was  that  of 
aggregation:  combining  in  meaningful  and  statistically  significant 
ways  the  substantial  number  of  variables  that  seemed  to  affect  computer 
programming  resource-costs. 

One  practical  application  for  an  aggregation  of  this  type  is  the  rather 
common  problem  of  ranking  programming  jobs  as  to  their  "costliness."  Managers 
often  want  to  compare  similar  tasks  to  determine  the  range  of  costs  that  can 
be  expected.  While  dollar  estimates  are  usually  available  and  have  the 
necessary  virtue  of  combining  many  variables  into  a  single  number,  the  result 
is  often  oversimplified,  i.e.,  many  managers  want  to  make  nonbudgetary 
comparisons  in  terms  of  resource-costs  such  as  man  months,  number  of  instruc¬ 
tions,  etc.,  rather  than  dollars.  Of  course,  the  programming  process  is 
complex  enough  that  a  single  resource-cost  is  not  a  sufficiently  meaningful 
yardstick  for  most  purposes.  For  this  reason,  we  were  seeking  a  dimensionless 
aggregate  of  resource-cost  variables  that  could  be  used  as  a  measure  of  the 
"costliness"  of  programming  jobs. 

The  correlation  analysis  clearly  indicated  that  two  variables.  Pages  of 
Documentation  and  the  Total  Number  of  Trips,  were  significantly  correlated 
with  the  major  dependent  variables,  i.e.,  they  tended  to  reinforce  the  same 
variance.  The  intercorrelation  matrix  for  the  primary  dependent  variables  is 
shown  in  Table  I. 


TABLE  I 

INTERCORRELATION  MATRIX  FOR  PRIMARY  DEPENDENT  VARIABLES 


Var. 

No. 

^O 

No. 

Trips 

Pages 

Doc. 

Total 

Man 

Months 

Total 

Comp. 

Hours 

New 

Instr. 

10 

Log^Q  Number  of  Trips 

1.00 

.51 

.62 

.61 

•55 

118 

Log^Q  Pages  of  Documentation 

•  51 

1.00 

.81 

.72 

.74 

1 

Log^Q  Total  Man  Months 

.62 

.81 

1.00 

.86 

00 

• 

9 

Log^  Total  Computer  Hours 

.61 

.72 

.86 

1.00 

.86 

20 

Log^  New  Instructions  (Machine) 

•  55 

.74 

.87 

.86 

1.00 
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This  interdependency  suggested  the  application  of  factor  analysis  techniques, 

1. e.,  to  the  definition  of  a  single  underlying  factor  in  terms  of  which  the 
variance  contributed  by  the  components  above  could  be  represented. 

Although  no  distinct  pattern  of  intercorrelations  was  apparent  among  the 
independent  variables,  we  hoped  that  factor  analysis  would  also  reveal  under¬ 
lying  relationships  that  would  permit  the  grouping  of  potential  predictors. 

2.  Factor  Analysis  Results.  The  first  25  principal  factors  derived  from  the 
analysis  of  69  dependent  and  independent  variables  were  studied  before  and 
after  rotation  (a  technique  to  sharpen  the  distinctions  between  significantly 
related  groupings).  The  relation  among  the  five  aspects  of  resource-cost 
mentioned  previously  was  delineated  in  the  first  factor,  which  is  shown  in 
Appendix  VII. 

Both  the  69  dependent  and  independent  variables,  and  the  40  independent 
variables,  were  analyzed  with  factor  analysis  techniques  for  groupings  among 
the  independent  variables.  The  results  were  rather  disappointing.  No  distinct 
and  interpretable  combinations  of  independent  variables  were  selected.  How¬ 
ever,  a  more  extensive  factor  analysis  and  the  application  of  more  sophisticated 
techniques  ( 12)  may  well  have  revealed  acceptable  predictor  factors.  Since 
time  was  limited,  a  different  approach  to  grouping  predictors  was  successful 
and  is  discussed  in  the  next  Section.  We  intend  to  consider  factor  analysis 
again  when  a  larger  data  base  is  available. 

3.  Development  of  the  Costliness  Factor.  To  obtain  proper  weighting,  the 
Costliness  Factor  was  regressed  against  its  five  components.  The  resultant 
regression  coefficients  (l4)  were  used  to  form  the  factor,  and  are  shown  in 
Appendix  VII.  The  validities  of  the  component  variables  with  the  Costliness 
Factor,  i.e.,  a  measure  of  the  degree  to  which  their  variance  is  represented, 
is  indicated  by  Table  II: 


TABLE  II 

VALIDITY  OF  THE  COMPONENTS  WITH  THE  COSTLINESS  FACTOR 


Var. 

No. 

Validity  of  Weighted 
Components  with  the 
Costliness  Factor, 
Variable  154 

10 

Log^0  Number  of  Trips 

•  73 

118 

Log10  Pages  of  Documentation 

.80 

1 

Log^0  Total  Man  Months 

.91 

9 

Log^Q  Total  Computer  Hours 

• 

00 

00 

20 

Log^Q  New  Instructions  (Machine) 

• 

00 

00 
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The  multiple  correlation  coefficient  between  all  of  the  weighted  components 
and  the  Costliness  Factor  is  0.9 6  out  of  a  maximum  of  1.00. 

The  frequency  distribution  of  "costliness"  for  the  74  data  points  is  shown 
in  Figure  9,  based  on  a  mean  of  100  and  a  standard  deviation  of  20. 


FIGURE  9 

FREQUENCY  DISTRIBUTION  OF  THE  COSTLINESS  FACTOR 


Note  that,  considering  the  lack  of  selection  criteria  for  the  data  base,  the 
distribution  is  rather  well  proportioned,  i.e.,  tends  to  unimodality  and 
symmetry . 
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SECTION  XV 


THE  DEVELOPMENT  OF  TASK  INDICES 


1.  Summary  of  Prior  Analytical  Steps.  The  development  of  indices  that 
describe  the  programming  task  is  presented  in  this  Section,  and  we  consider 
these  groupings  as  the  major  analytical  finding  of  the  analysis.  However, 
the  delineation  of  meaningful  aggregates  of  variables  was  dependent  on  the 
availability  of  a  working  set  of  predictor  variables,  i.e.,  one  from  which 
spuriousity  and  redundancy  have  been  removed.  This  required  the  preparatory 
analytical  steps  discussed  in  Sections  IX  through  XIV. 

First,  the  questionnaires  were  studied  for  accuracy  and  completeness.  Suspicious 
(or  nonexistent)  responses  were  discussed  with  the  programmer  to  resolve 
misinterpretations  whenever  possible.  In  about  one  percent  of  the  cases, 
values  were  estimated  by  the  Project  staff.  Since  data  were  not  evenly 
distributed  for  many  variables,  i.e.,  there  were  many  smaller  values  close 
together,  and  fewer  large  values  rather  widely  separated,  logarathmic 
transformations  were  applied  in  many  instances  to  achieve  a  more  continuous 
and  more  normal  array. 

Next,  scatterplot  and  correlation  analysis  were  used  to  study  the  relation¬ 
ships  among  the  variables.  Those  which  had  low  validities,  i.e.,  were  not 
strongly  related  to  the  dependent  cost  variables,  were  considered  for  elimi¬ 
nation,  as  were  those  predictors  that  tended  to  duplicate  each  other's 
contribution  to  prediction.  Factor  analysis  was  then  applied  in  an  attempt 
to  obtain  groupings  of  variables.  A  nondimensional  aggregate  of  related 
aspects  of  re source- costs  was  defined,  and  termed  "Costliness."  However,  the 
factor  analysis  procedures  failed  to  delineate  any  distinct  and  interpretable 
groupings  of  predictors. 

As  a  result,  we  tried  another  approach  to  obtain  combinations  of  independent 
variables.  This  involved  what  might  be  thought  of  as  the  opposite  of  factor 
analysis:  the  grouping  of  variables  that  contributed  intuitively  similar 
but  statistically  different  ingredients  to  the  prediction.  Initially,  the  1*0 
independent  variables  were  divided  into  5  intuitively  attractive  groupings 
that  corresponded  to  different  influences  on  computer  programming  costs.  The 
components  in  each  group  with  relatively  high  intercorrelations  and  low 
validities  with  the  dependent  variables  were  then  weeded  out  in  favor  of  those 
predictors  with  the  lowest  intercorrelations  and  the  highest  validities.  The 
development  of  each  of  the  indices  is  considered  separately  in  this  context. 
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2.  The  Uniqueness  Index.  Ten  of  the  40  independent  variables  shared  the 
characteristic  of  "being  rather  uncommon,  yet  having  a  substantial  impact  on 
program  cost  when  present.  These  variables  were  considered  for  a  measure  of 
the  uniqueness  of  the  task.  The  initial  correlation  and  validity  relation¬ 
ships  for  the  10  variables  evaluated  are  shown  in  Table  III.  Note  that  only 
the  validities  with  the  Costliness  Factor  are  shown;  however,  validities  with 
each  of  the  dependent  variables,  i.e.,  Man  Months,  Computer  Hours,  Number  of 
New  Instructions,  and  Months  Elapsed,  were  also  considered  before  deletions 
were  made. 

The  intent  of  the  winnowing  process  was  to  obtain  a  few  predictors  with 
significant  cost  validities,  i.e.,  predictive  power  and  relatively  low  inter¬ 
correlations,  i.e.,  redundancy.  The  resulting  four  components  are  shown  in 
Table  IV.  Note  that  all  of  the  predictors  in  Table  IV  are  two- valued,  i.e., 
always  either  one  or  zero,  so  that,  when  uniqueness  as  defined  is  not  present, 
the  index  adds  zero  to  the  overall  prediction. 

To  weight  the  predictors  properly  in  terms  of  their  contribution  to  prediction, 
each  of  the  four  dependent  variables,  and  the  Costliness  Factor,  were  separately 
regressed  against  the  four  components  of  the  Uniqueness  Index.  The  raw 
regression  coefficients  for  each  variable  were  averaged,  and  rescaled,  to 
obtain  the  following  weights^: 

1.2  x  Innovation  in  System 

1.0  x  Stringent  Timing  Requirements 

1.7  x  First  Programming  Effort  on  Computer 

1.3  x  Program  Developed  at  More  Than  One  Location 

The  Uniqueness  Index  was  defined  as  the  sum  of  the  weighted  variables  as 
shown  and  is  identified  for  this  analysis  as  variable  1^9 .  The  maximum  value 
for  this  index  is  5.2;  the  minimum  zero,  and  its  validity  with  the  Costliness 
Factor  is  O.52.  (Note  that  the  validity  of  the  weighted  and  summed  ingredients 
of  the  index  is  considerably  greater  than  for  any  of  the  components  taken 
separately . ) 

The  distribution  of  the  Uniqueness  Index  for  the  7^  data  points  is  shown  in 
Figure  10,  based  on  a  mean  of  100  and  a  standard  deviation  of  20. 


O 

°The  details  of  the  regression  and  weighting  process  for  this  index  are 
provided  in  Appendix  IX. 
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CORRELATION  MATRIX  OF  VARIABLES  CONSIDERED  FOR  THE  UNIQUENESS  INDEX 
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106  Did  Program  Development 
Take  Place  at  More  Than 
One  Location 


TABLE  IV 
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FIGURE  10 

FREQUENCY  DISTRIBUTION  OF  THE  UNIQUENESS  INDEX 

3.  The  Job  Difficulty  Index.  Eight  of  the  40  predictor  variables  were 
intuitively  grouped  into  a  measure  of  the  difficulty  of  the  programming  job. 
The  intercorrelations  and  illustrative  validities  with  the  Costliness  Factor 
for  these  8  variables  is  shown  in  Table  V. 

The  intercorrelations  and  validities  for  the  predictor  variables  selected  as 
components  of  the  Job  Difficulty  Index  are  as  follows: 

TABLE  VI 

CORRELATION  MATRIX  OF  THE  VARIABLES  SELECTED  FOR  THE  JOB  DIFFICULTY  INDEX 


Var. 

Logio  No. 

Logio  No. 

Validity  With 

No. 

Subprog . 

Classes  in  D.B. 

Costliness  Factor 

30 

Log10  Number  of 
Subprograms 

1.00 

•13 

.57 

32 

Log10  Number  of  Classes 
in  Data  Base 

•13 

1.00 

.22 
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lTION  matrix  of  the  variables  considered  for  the  job  difficulty  index 
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As  before,  the  four  dependent  variables,  and  the  Costliness  Factor  were 
regressed  against  the  two  components  of  the  Job  Difficulty  Index,  and  averaged 
and  rescaled  coefficients  used  to  obtain  proper  weighting.  The  final  form  of 
the  Index  was  defined  as  the  sum  of  the  weighted  variables  as  shown: 

5*0  x  Log-^0  Number  of  Subprograms 

1.0  x  Log1Q  Number  of  Classes  in  Data  Base 

Note  that,  due  to  the  logarithmic  transformations,  the  addition  of  weighted 
variables  really  represents  the  multiplication  of  power  functions  of  the  raw 
data.  The  Job  Difficulty  Index  has  a  lower  limit  of  zero,  and  no  upper  limit, 
and  is  identified  in  this  analysis  as  variable  150.  The  reader  should  note 
that  any  program  in  its  entirety,  is  considered  to  represent  at  least  one 
subprogram  and  one  class  of  data.  So,  in  computing  this  index,  the  recorded 
values  of  the  two  raw  components  are  always  increased  by  one.  Since  the  log,n 
of  one  is  zero,  the  lower  limit  for  the  index  occurs  when  the  recorded  values 
of  the  raw  components  are  zero  and  the  entire  program  is  considered  to  be 
one  subprogram  and  one  class  of  data. 


FIGURE  11 

FREQUENCY  DISTRIBUTION  OF  THE  JOB  DIFFICULTY  INDEX 
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4.  Development  Environment  Index.  Seven  of  the  40  predictor  variables  were 
intuitively  grouped  into  a  measure  of  the  development  environment .  The  inter- 
correlations  and  illustrative  validities  of  these  7  variables  are  shown  in 
Table  VII. 

The  intercorrelations  and  validities  for  the  predictors  selected  as  components 
of  the  Development  Environment  Index  are  as  follows: 

TABLE  VIII 

CORRELATION  MATRIX  OF  THE  VARIABLES 
SELECTED  FOR  THE  DEVELOPMENT  ENVIRONMENT  INDEX 


Var . 

Est .  Cust . 

Prog. 

Validity  With 

No. 

Exper . 

In  Des. 

Costliness  Factor 

100 

Estimate  Customer  Experience 

1.00 

.04 

-  .21 

1^3 

Percent  Programmers  Partici¬ 
pating  in  Design 

.04 

1.00 

-  .49 

Again,  the  four  dependent  variables,  and  the  Costliness  Factor,  were  regressed 
against  the  predictors,  and  averaged  and  rescaled  coefficients  used  to  obtain 
weights . 

The  final  form  of  the  Development  Environment  Index  was  defined  as  the  sum  of 
the  weighted  variables  as  shown: 

-1.0  x  Estimate  Customer  Experience 

-3.4  x  Percent  Programmers  Participating  in  Design 

The  negative  weights  were  selected  so  that  the  regression  coefficient  of  the 
Index  would  be  positive,  i.e.,  increased  values  would  result  in  higher  costs, 
which  is  more  consistent  with  the  weighting  of  the  other  indices. 

The  Development  Environment  Index  has  a  lower  limit  of  -1.0  and  an  upper 
limit  of  -6.4,  and  is  identified  as  variable  151  in  this  analysis.  Its 
validity  with  the  Costliness  Factor  is  0-53-  The  frequency  distribution, 
based  on  a  mean  of  100  and  a  standard  deviation  of  20,  is  shown  in  Figure  12. 
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CORRELATION  MATRIX  OF  THE  VARIABLES  CONSIDERED  FOR  THE  DEVELOPMENT  ENVIRONMENT  INDEX 
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20-1 


N  -  74  Computer  Programs 
Mean  =  100 

Standard  Deviation  =  20 


FIGURE  12 

FREQUENCY  DISTRIBUTION  OF  THE  DEVELOPMENT  ENVIRONMENT  INDEX 


5 •  Job  Type  Index.  Nine  of  the  40  predictor  variables  vere  intuitively 
grouped  into  a  measure  of  the  types  of  data-handling  that  were  characteristic 
of  the  programming  job.  The  intercorrelations  and.  illustrative  validities 
for  these  9  variables  are  shown  in  Table  IX. 

The  intercorrelations  and  validities  for  the  variables  selected  as  components 
of  the  Job  Type  Index  are  as  follows: 

TABLE  X 

CORRELATION  MATRIX  OF  THE  VARIABLES  SELECTED  FOR  THE  JOB  TYPE  INDEX 


Var. 

No. 

Clerical 

$  Trans. 

&  Ref. 

* 

Gener. 

Valid.  With 
Costl.  Factor 

37 

Percent  Clerical 
Instructions 

1.00 

•  17 

-  .14 

-  .23 

46 

Percent  Transformation  and 
Reformatting  Function 

.17 

1.00 

.08 

-  .13 

47 

Percent  Generation  Function 

-  .14 

.08 

1.00 

.29 
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CORRELATION  MATRIX  OF  THE  VARIABLES  CONSIDERED  FOR  THE  JOB  TYPE  INDEX 
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The  4  dependent  variables,  and  the  Costliness  Factor,  were  again  regressed 
against  the  3  components  of  the  Index,  and  averaged  and  rescaled  coefficients 
used  to  obtain  weights.  The  final  form  of  the  Job  Type  Index  was  defined  as 
the  sum  of  the  weighted  variables  as  shown: 

-  1.0  x  Percent  Clerical  Instructions 

-  1.3  x  Percent  Transformation-Reformatting  Function 
4.4  x  Percent  Generation  Function 

The  Job  Type  Index  has  a  lower  limit  of  -  1.3  and  an  upper  limit  of  4.4,  and 
is  identified  in  this  analysis  as  variable  152.  Its  validity  with  the  Costli¬ 
ness  Factor  is  0.45.  The  frequency  distribution,  based  on  a  mean  of  100  and 
a  standard  deviation  of  20  is  shown  in  Figure  13. 


FIGURE  13 

FREQUENCY  DISTRIBUTION  OF  THE  JOB  TYPE  INDEX 
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6.  Staffing  Index.  Several  attempts  were  made  during  the  analysis  to 
define  a  useful  measure  of  the  experience  and  capability  of  the  programming 
staff.  None  of  these  attempts  demonstrated  statistical  significance. 

We  intend  to  pursue  this  matter  further,  especially  since  a  measure  of  staff 
capability  has  such  a  high  intuitive  appeal  as  a  predictor  of  programming 
costs. 
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SECTION  XVI 


AN  EVALUATION  OF  THE  TASK  INDICES 


As  we  have  noted  previously,  the  main  appeal  of  the  task  indices  is  not  so 
much  their  statistical  significance  as  their  interpretability.  After  the 
winnowing  process  discussed  in  Section  XV  and  the  resulting  eleven  predictor 
variables  could  have  been  directly  combined  into  regression  equations  along 
the  lines  of  those  discussed  in  Section  XIII,  i.e.,  without  grouping.  While  the 
presence  of  more  than  four  or  five  predictors  in  a  single  linear  regression 
equation  is  rather  unusual,  and  could  lead  to  undesirable  weighting,  the 
statistical  power  would  likely  not  be  a  great  deal  different  than  if  the  task 
indices  had  been  used.  In  other  words,  the  relationships  between  the  eleven 
independent  and  the  dependent  variables  are  what  underly  the  prediction, 
whether  the  independents  are  weighed  and  grouped  into  indices  or  not.  In 
this  sense,  the  indices  reflect  our  desire  to  assure  that  a  more  comprehensive 
set  of  predictors  are  represented  in  the  equation,  and  suitably  weighted,  rather 
than  limit  the  number  of  predictors  to  four  or  five  (and  run  the  risk  of 
oversimplification) . 

Moreover,  the  indices  facilitate  the  comparison  of  programming  jobs,  e.g., 
whether  one  is  more  unique,  or  difficult  than  another,  and  the  planning  that 
managers  do  to  reduce  costs,  e.g.,  an  increase  in  programmer  participation  in 
program  design  to  lessen  the  contribution  to  cost  of  the  Development  Environ¬ 
ment  Index. 

It  is  clear,  however,  that  the  four  indices  developed  in  this  analysis  are 
rather  rudimentary.  While  each  of  the  eleven  components  tend  to  represent 
the  contribution  of  many  other  variables,  i.e.,  each  of  the  eleven  is  related 
to  others  that  are  not  components  of  an  index,  there  are  other  aspects  of 
the  programming  process  that  are  omitted.  We  intend  to  refine  and  augment 
the  present  indices  in  subsequent  analyses  of  this  series. 
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SECTION  XVII 


THE  ESTIMATION  OF  DEPENDENT  COST  VARIABLES  USING  TASK  INDICES 


1.  General  Approach.  We  are  now  ready  to  translate  the  results  presented  in 
Sections  IX  through  XVI  into  a  management  tool  for  estimating  the  primary 
resource-cost  variables,  i.e.,  Man  Months,  Computer  Hours,  Months  Elapsed, 
and  New  Instructions,  and  the  Costliness  Factor.  To  obtain  regression 
estimation  equations,  each  of  these  dependent  variables  was  separately 
regressed  against  the  task  indices  developed  in  Section  XVT.  The  components 
of  each  index  are  summarized  in  Table  X. 

q 

The  resultant  regression  equations  are  shown  in  Table  XI  .  The  technical 
details  of  the  regressions  are  provided  in  Appendix  X. 

2.  Development  of  the  Stanine  Plots.  It  is  customary  in  statistical 
estimation  to  construct  confidence  bands  that  straddle  the  prediction  function 
and  to  establish  the  limits  within  which  a  certain  proportion  of  all  observed 
values  will  fall,  for  equally-sized  samples-*-®.  Although  the  confidence- 
interval  approach  provides  a  useful  safeguard  against  unwarranted  reliance 

on  computed  estimates,  single  bands  straddling  the  prediction  line  are  too 
inflexible  for  estimating  computer  programming  costs.  The  reason  is  that 
the  values  near  each  extreme  of  the  band  have  a  much  lower  probability  of 
occurring  than  those  near  the  center,  i.e.,  observations  tend  to  be  normally 
distributed  around  each  estimate. 


9 

The  computed  prediction  function  that  minimized  the  least-squares  variance 
of  the  observed  values  tended  to  provide  overestimates  of  smaller  programs 
and  underestimates  of  larger  programs.  (This  was  due  in  part  to  the  fact 
that  the  constant  in  the  regression  equation  tends  to  inflate  smaller 
estimates,  and  in  part  to  the  nature  of  the  least-squares  procedure.)  For 
this  reason,  the  function  was  rotated  slightly  to  equalize  the  variance  of 
the  estimated  and  observed  points.  The  effect  of  this  adjustment  was  to 
distribute  the  positive  and  negative  estimation  variances  more  equally 
along  the  entire  range  of  interest.  The  loss  in  predictive  efficiency  was 
less  them  two  percent. 

10 In  principle,  this  means  that  the  0.95  confidence  beind  will  encompass 
95  percent  of  all  observations  based  on  a  given  estimate,  for  equally- 
sized  samples. 
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COMPONENTS  OF  THE  TASK  INDICES 
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NOTE:  In  the  computation  for  the  Job  Difficulty  Index,  the  values  of  each  raw  component 
should  be  increased  by  one,  since  any  programming  task,  i.e.,  data  point  in  its 
entirety,  is  considered  to  represent  a  difficulty  equal  to  at  least  one  subprogram 
and  one  class  of  data. 


COST  ESTIMATION  EQUATIONS 
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To  provide  the  user  of  the  cost  estimation  plots  with  a  more  flexible  means 
of  taking  the  probability  distribution  of  programming  tasks  into  account  and 
blending  in  his  own  experience,  a  novel  confidence-band  rationale  based  on  an 
application  of  the  Stanine  scoring  approach  was  used(20). 

Nine  confidence  bands  were  constructed,  with  the  center  band  straddling  the 
prediction  line  and  four  bands  on  each  side.  The  two  outer  bands  (l  and  9) 
are  open-ended.  All  the  other  bands  (2  through  8)  are  of  equal  width.  The 
probability  of  observed  values  falling  into  each  of  9  Stanine  confidence  bands 
is  as  follows: 


Probability 
of  Occurrence 


Stanine 

Band 


.04 

.07 

.12 

.17 

.20 

.17 

.12 

.07 

.04 


1 

2 

3 

4 

5 

6 

7 

8 

9 


Note  that  the  middle,  or  "5"  band,  represents  the  most  probable  or  expected 
value.  As  the  bands  differ  from  the  "5"  band,  they  contain  values  that  are 
less  and  less  likely  to  occur.  The  application  of  these  bands  to  estimation 
is  illustrated  best  with  an  example.  Consider  Figure  l4,  where  estimated 
versus  actual  values  for  the  74  points  in  the  data  base  axe  plotted  for  the 
Costliness  Factor.  The  middle  of  the  center,  or  "5"  band,  is  equivalent  to 
the  expected  resource-cost  computed  from  the  regression  equation.  The  limits 
of  the  "5"  band  define  the  variation  around  the  computed  estimate  for  the 
average  case.  The  "6"  through  "9"  bands  define  progressively  higher  variations 
from  the  computed  estimate;  the  "4"  through  "1"  bands  define  progressively 
lower  variations.  Note  that  the  points  tend  to  be  clustered  along  the 
prediction  line,  which  would  pass  through  the  middle  of  the  "5"  band.  In 
fact,  this  tendency  exceeds  the  theoretical  proportion.  For  instance, 
nearly  27  percent  of  the  points  fall  into  the  "5"  band,  more  than  the  20 
percent  expected.  This  effect  is  due  to  the  nature  of  the  plot,  which 
depicts  after-the-fact  relationships.  If  estimates  were  made  with  new  data, 
and  not  from  the  74-point  base  used  to  derive  the  equations,  we  would  expect 
the  distribution  of  observations  around  the  estimates  to  be  closer  to  the 
theoretical  values. 
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The  center,  or  "5”  band,  is  equivalent  to  a  0.20  confidence  estimation  for 
the  Costliness  Factor.  The  "4,"  "5,"  and  "6"  bands  together  are  equivalent 
to  a  0.5^  confidence  estimation  (.17  +  .20  +  .17)*  Higher-valued  confidence 
bands  can  be  constructed  simply  by  summing  the  appropriate  probabilities  of 
occurrence. 

3.  Application  of  the  Plots.  Suppose  that  a  manager  felt  that  a  proposed 
programming  task,  when  compared  to  the  spectrum  of  possible  programs,  repre¬ 
sented  an  "average"  type  of  job,  i.e.,  a  costliness  of  100.  What  is  the 
probability  that  costliness  scores  of,  say,  130  could  result?  By  following 
the  130  line  horizontally  to  the  vertical  100  line,  we  see  that  the  inter¬ 
section  falls  in  the  high  "9"  band.  If  the  manager's  intuitive  evaluation  of 
the  task  as  "average"  was  correct,  a  costliness  score  of  130  is  quite  unusual; 
i.e.,  it  would  occur  less  than  4  percent  of  the  time.  In  retrospect,  a  program 
product  that  scores  130  in  costliness  is  quite  unusual  from  any  point  of  vlerw 
(less  than  four  percent  of  all  jobs  will  score  130  or  higher),  regardless  of 
the  manager's  a  priori  evaluation.  The  Stanine  plot  for  costliness  thus 
provides  the  manager  with  a  tool  for  establishing  the  limits  of  variation 
to  be  expected  around  an  estimate  for  various  degrees  of  confidence. 

The  Stanine  plots  can  be  useful  in  other  ways  when  one  of  the  other  dependent 
variables  is  involved  (see  Figures  15  through  18  which  follow).  Consider 
Figure  15,  for  example,  which  depicts  estimated  versus  actual  values  for  man 
months.  Suppose  that,  based  on  the  appropriate  equation  in  Table  XII,  a 
user  computes  an  estimate  of  100  man  months  for  a  particular  programming  job 
(i.e.,  a  value  of  2.0  in  logarithms).  How  much  of  a  variance  around  this 
estimate  should  be  expected?  Several  options  are  available: 

a.  An  "average"  estimate  for  man  months,  based  on  a  computed  value  of 
100,  would  involve  confidence-band  limits  of  73  and  I30H.  The  probability 
of  values  occurring  in  this  range  is  at  least  0.20. 

b.  Based  solely  on  his  intuitive  appraisal  of  the  task,  the  components 
of  the  prediction  equation,  his  experience,  or  any  other  factors  pertinent  to 
the  problem,  the  user  can  consider  the  programming  job  as  equivalent  to  a 
higher  or  lower  confidence  band,  and  read  the  upper  and  lower  estimates 
directly.  A  score  of  "6, "  for  example,  would  imply  a  program  slightly  above 
average,  and  would  yield  confidence  band  limits  of  130  and  230  man  months. 

The  probability  of  values  occurring  in  this  range  is  at  least  0.17. 


i:LThe  limits  axe  not  symmetrical  around  the  estimate  due  to  the  use  of 
logarithmical .ly- transformed  variables. 
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c.  To  hedge  his  estimation,  the  user  can  sum  the  probabilities  of  occur¬ 
rence  from  different  bands.  For  example,  the  0. 5k  confidence  limits  for  100 
man  months  would  be  40  and  230  man  months.  Another  approach  would  be  to 
assume  that  errors  due  to  underestimation  are  more  likely  than  overestimation. 

In  this  case,  the  "5"-  and  ”6"-band  limits  of  73  and.  230  man  months  would  be 
expected  to  contain  twice  the  usual  proportion  of  observed  values  (i.e.,  at  ^ 
least  54  percent),  since  occurrences  in  bands  "4"  or  lower  are  not  considered. 

The  wideness  of  the  limits  is  due  mainly  to  two  causes:  first,  the  fact  that 
all  the  types  of  programs  represented  in  the  data  base  were  analyzed  together, 
i.e.,  no  subpopulations  were  considered  separately.  We  intend  to  explore  the 
effects  of  subpopulations  in  subsequent  analyses,  and  hope  to  tighten  the 
confidence-band  limits  as  a  result.  The  second  reason  for  the  rather  wide 
bands  is  due  to  more  fundamental  matters  involving  the  combination  of  before 
and  after-the-fact  probabilities.  This  question  is  discussed  in  the  following 
Section. 

The  use  of  Figures  16,  17,  and  18,  dealing  with  computer  hours,  new  instructions, 
and  months  elapsed,  is  similar. *3 


12 

These  approaches  are  comparable  in  principle  to  the  one-  and  two-tail  tests 
common  to  statistical  inference. 

13We  would  like  to  point  out  that  the  straight-line  confidence  bands  shown  in 
Figures  14  through  18  are  approximations.  The  bands  actually  diverge,  as 
the  variables  involved  in  the  estimation  depart  from  their  mean  and  the 
prediction  line,  i.e.,  the  corrections,  are  added  to  the  estimate  above  the 
line,  and  subtracted  from  it  below  the  line.  The  corrections  are  unique  for 
each  value  of  each  predictor,  however,  and  rather  laborious.  A  worksheet 
is  provided  in  Appendix  XI  for  readers  who  wish  to  compute  the  corrections 
for  each  application  of  the  estimating  equations.  Two  examples  worked  out 
in  the  Appendix  illustrate  the  magnitude  of  the  corrections,  which  are  +13 
percent  for  the  "2"  and  "8"  bands. 
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SECTION  XVIII 


AN  EVALUATION  OF  THE  STANINE  COST  ESTIMATION  RELATIONSHIPS 


As  we  have  observed  earlier*  we  recommend  the  use  of  the  cost  estimation 
equations  and  Stanine  plots  as  guides  and  checks  for  the  manager  of  computer 
programming.  In  no  sense  are  they  yet  sufficiently  adequate  or  reliable 
techniques  for  the  prediction  of  resource-costs  in  terms  of  a  particular 
job.  There  are  several  reasons  why  this  is  so. 

First,  the  limits  of  the  confidence  bands  are  still  too  far  apart  to  be 
useful  to  a  manager  for  prediction.  Based  on  "5"  band,  or  "average"  task, 

Figure  15  indicates  that  the  range  of  values  that  can  be  expected  around  an 
estimate  of  100  man  months  (a  computed  logarithm  of  2.0)  lies  between  73  and. 

130  man  months.  This  probably  represents  close  to  the  widest  interval  in  which 
a  manager  would  be  interested,  i  .e .,  about  plus  or  minus  30  percent •  The 
addition  of  more  bands  would,  of  course,  result  in  even  more  disparate  limits. 

The  very  wideness  of  the  confidence  intervals,  which  lessens  the  value  of  the 
plots  for  prediction,  also  increases  their  usefulness  as  a  check  on  estimates 
made  by  other  means,  i.e.,  values  that  fall  outside  the  expected  bands  are 
especially  suspicious.  For  a  relatively  small,  simple  programming  task,  an 
estimate  that  falls  into  the  high  "7"  band  when  compared  to  the  value  computed 
from  the  equations  would  reasonably  be  subject  to  review. 

Similarly,  for  a  programming  task  that  is  expected  to  be  sizeable  and  complex, 
costliness  estimates  that  fall  into  the  low  "3"  or  "2"  bands  when  compared  to 
the  estimates  computed  from  the  equation  would  quite  rightly  be  considered  as 
unusual,  and  subjected  to  further  scrutiny. 

The  aim  of  future  research  in  this  area  is:  (l)  to  broaden  the  size  and 
representativeness  of  the  data  base  from  which  the  estimating  relationships 
are  derived;  (2)  to  define  and  study  subpopulations  separately;  (3)  to  improve 
our  understanding  of  the  programming  process;  (4)  by  all  of  these  approaches, 
to  narrow  the  range  of  expected  variation  around  a  computed  estimate.  In  this 
context,  the  current  collections  of  data  from  various  government  and  industry 
sources  are  a  step  in  the  right  direction. 

A  more  substantial  problem  for  future  research  is  to  find  a  more  reliable 
basis  for  blending  the  manager's  intuitive  and  experiential  estimates  with  the 
statistical  results.  The  Stanine  plots  certainly  represent  a  practical  approach 
that  we  believe  is  more  suited  to  general  application  than  the  techniques  that 
we  have  used  previously.  But  the  so-called  "probability  of  occurrence" 
associated  with  each  Stanine  band  (0.20  for  the  "5"  band,  0.17  for  the  "4"  and 
"6"  bands,  etc.)  reflects  the  proportion  of  all  computer  programming  jobs  that 
likely  will  fall  into  each  band.  That  is  not  what  the  manager  is  interested 
in.  He  wants  to  know  the  probability  that,  for  a  particular  program,  the 
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final  resource-costs  will  fall  within  the  limits  prescribed  by  a  given  Stanine 
band,  or  a  combination  of  bands.  (In  somewhat  this  sense,  the  Stanine  proba¬ 
bility  of  occurrence"  represents  a  lower  limit.) 

If  a  manager  considers  a  programming  job  to  be  "average,"  and  if  he  is  right, 
the  probability  that  the  resource-costs  will  fall  within  the  limits  denoted 
by  the  "5"  band  exceeds  the  0.20  minimum,  and  could  go  as  high  as  1.00. 

What  is  needed  to  make  the  probabilities  more  realistic  is  a  defensible  way 
of  combining  a  manager's  subjective  estimation  of  the  band  within  which  a  job 
will  fall  with  the  theoretical  probability  of  occurrence. 

At  the  present  state-of-the-art,  we  feel  that,  unless  a  manager  has  strong 
indications  that  a  programming  task  is  unusually  simple  or  difficult,  the  "5" 
band  values  should  be  used.  We  also  expect  that  for  most  programs,  the  chances 
of  obtaining  results  within  the  "5"  band  limits  are  one  in  two,  or  better. 

But  we  have  no  data  by  which  these  assertions  can  be  defended  and  by  which, 
for  instance,  a  manager's  subjective  confidence  of  0.80  that  a  job  is  properly 
represented  in  the  "5"  band  can  be  reliably  combined  with  the  0.20  theoretical 
minimum  probability  of  occurrence  to  obtain  a  compound  probability  that  ultimate 
program  costs  will  fall  within  the  prescribed  limits. 

The  long-term  alleviation  of  this  problem  lies  mainly  in  the  on-line  data 
collection  that  we  will  be  undertaking.  In  addition  to  obtaining  data  about 
the  task,  the  environment,  the  staff,  the  resources  and  expenditures,  etc., 
for  a  sample  of  actual  programming  jobs,  we  intend  to  record  the  manager's 
a  priori  estimate  of  the  Stanine  band  that  he  considers  appropriate,  and,  by 
comparing  it  with  the  after-the-fact  costs,  derive  an  empirical  function  that 
represents  the  degree  to  which  subjective  estimates  should  be  considered,  i.e., 
weighted,  for  different  types  of  program  products. 

Eventually  (and  this  will  take  longer),  we  hope  to  use  the  on-line  data  to 
derive  relationships  that  can  be  used  to  correct  a  priori  estimates  on  the 
basis  of  actual  costs  to  date .  This  approach  will  carry  us  into  considerations 
of  what  are  called  Bayesian  inferences,  i.e.,  procedures  with  prior  and  pos¬ 
terior  probabilities  that  permit  the  use  of  the  latest  factual  information  to 
correct  earlier  estimates.  A  good  deal  of  data  collection  and  analysis  lies 
between  our  present  capability  and  understanding  and  the  realization  of  the 
potential  that  is  inherent  in  the  Stanine  plots.  Our  plans  into  beyond 
the  on-line  data  collection  and  analysis,  however,  are  deliberately  aimed 
toward  the  development  of  these  and  similar  techniques  for  management  use  in 
computer  programming. 
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SECTION  XIX 


THE  SENSITIVITY  OF  PROGRAMMING  RESOURCE-COSTS 
TO  CHANGES  IN  THE  TASK  INDICES 


1.  Background.  The  equations  that  were  developed  in  the  previous  Section 
have  two  uses:  the  most  important,  of  course,  is  the  estimation  of  resource- 
costs  for  programming  jobs.  In  addition,  the  weights  that  are  given  each 
index  may  be  used  to  infer  their  relative  impact  on  the  dependent  variable. 

This  would  guide  the  programming  manager's  decisions  as  to  planning  trade-offs, 
e.g.,  to  reduce  total  resource-costs,  is  it  more  advantageous  to  reduce  the 
"difficulty"  of  the  job  or  improve  the  "development  environment, "  as  we 
measure  them? 

Unfortunately,  the  equation  coefficients  for  each  index  are  not  susceptible  to 
direct  interpretation.  One  reason  is  that  some  of  the  predictors  that  enter 
into  the  indices  are  in  raw  form  while  others  are  logarithms  or  percentages. 

The  weight  given  to  each  index  tends  therefore  to  reflect  the  importance  of 
the  combination  of  the  variables  as  they  appear  in  the  equation,  and  not  as 
the  manager  considers  them,  i.e.,  in  raw  form. 

Another  obstacle  to  direct  interpretation  of  the  regression  coefficients  is 
that  some  of  the  indices  vary  through  a  much  wider  range  than  others.  An 
index  with  a  relatively  low  weight  but  a  large  variance  may  exert  a  greater 
impact  on  resource-costs  than  an  index  with  a  high  weight  and  little  variation. 
For  instance,  the  Uniqueness  Index,  which  is  composed  of  five  binary  predictors, 
varies  through  a  much  smaller  range  than  does  the  Job  Difficulty  Index,  which 
has  no  upper  limit. 

2.  The  Limitations  of  this  Approach.  A  formal  sensitivity  analysis (21) 
requires  a  great  deal  more  data  than  are  available  in  our  present  Jk- point 
base.  In  this  sense,  the  material  in  this  Section  is  exploratory,  and  is 
included  to  illustrate  the  application  of  the  sensitivity  concept.  We 
expect  to  develop  more  trenchant  and  reliable  measures  of  the  relative 
impact  of  the  indices  in  subsequent  analyses,  as  the  data  base  is  enlarged 
and  refined. 

3.  The  Analytical  Procedures  and  Results.  To  determine  the  sensitivity  of 
each  of  three  resource-cost  variables,  Man  Months,  Computer  Hours,  and  Months 
Elapsed,  values  for  them  were  computed  from  the  estimation  equations,  with 
each  of  the  indices  at  their  respective  means.  Next,  for  each  equation  (i.e., 
dependent  variable)  at  a  time,  the  indices  were  iteratively  changed  one  at  a 
time  while  the  remaining  three  indices  were  held  at  their  means.  Four  vari¬ 
ations  were  applied  to  each  index:  +10  percent,  +1  standard  deviation,  -10 
percent,  and  -1  standard  deviation.  The  difference  between  the  effect  on 
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resource-costs  of  the  10-percent  and  1-standard-deviation  changes  is  intended 
to  illustrate  the  impact  of  the  wide  variations  that  are  possible  in  certain 
indices . 

The  results  of  these  procedures  are  illustrated  in  Figures  19,  20,  and  21. 

The  actual  values  are  provided  in  tabular  form  in  Appendix  XII. 

4.  Interpretation  of  the  Results.  Several  general  conclusions  can  be  made 
from  a  consideration  of  Figures  19,  20,  and  21: 

a.  The  Uniqueness  and  Job  Difficulty  Indices  have  a  very  substantial 
impact  on  the  estimation  for  Total  Computer  Hours.  Increasing  either  index 
by  1  standard  deviation  from  its  mean  essentially  doubles  the  estimate  for 
this  resource-cost.  (Reducing  the  Uniqueness  and  Job  Difficulty  Indices  by 
1  standard  deviation  from  their  mean  reduces  the  estimate  for  Computer  Hours 
by  about  one-half.  The  asymmetry  is  due  to  the  nature  of  the  data  that  were 
used  in  the  analysis.)  Increasing  or  decreasing  the  Uniqueness  and  Job 
Difficulty  Indices  by  1  standard  deviation  from  their  means  has  a  lesser 

but  still  significant  impact  on  the  estimates  for  Total  Man  Months  and  Months 
Elapsed,  e.g.,  changes  in  the  estimate  from  22  to  68  percent  result. 

b.  The  Development  Environment  Index  has  a  considerable  effect  on  the 
estimation  for  Total  Man  Months  and  Total  Computer  Hours .  Changes  of  10 
percent  from  the  mean  in  this  index  result  in  estimate  changes  of  from  15  to 
19  percent  in  the  dependent  variables.  Changes  of  1  standard  deviation  from 
the  mean  result  in  estimate  changes  of  from  43  to  83  percent  in  Total.  Man 
Months  and  Total  Computer  Hours. 

c.  Small  changes  in  the  Job  Type  Index  have  little  impact  on  estimates 
for  the  three  resource-cost  variables  under  consideration.  Due  to  its  wide 
variance,  however,  changes  in  this  index  of  1  standard  deviation  from  the 
mean  cause  changes  in  the  estimates  of  from  13  to  66  percent  in  the  dependent 
variables . 

d.  Months  Elapsed  is  about  one-half  as  sensitive  to  changes  in  the 
indices  as  Total  Man  Months  and  Total  Computer  Hours.  This  may  be  a 
reflection  of  the  fact  that  managers  tend  to  satisfy  schedule  constraints 
by  making  compensatory  changes  in  Total  Man  Months  and  Total  Computer  Hours. 


Note  that  the  Development  Environment  Index  (variable  151)  decreases  as 
resource-costs  increase,  and  vice  versa.  This  occurs  because  the  components 
of  the  index  are  negatively  weighted  to  be  consistent  with  intuitive  expec¬ 
tation,  i.e.,  so  that  improvements  in  the  environment  (larger  negative 
index  values)  decrease  resource-costs. 
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More  Data 
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NOTE:<7"=  Standard  Deviation 


FIGURE  19 

PERCENT  CHANGE  IN  MAN  MONTHS 
DUE  TO  INDICATED  CHANGE  FROM  THE  INDEX  MEAN 


89 


COMPUTER  HOURS  COMPUTER  HOURS 


Improved 
]  Development 
Environment 


DEVELOPMENT  ENVIRONMENT 


NOTE:  cr  =  Standard  Deviation 


□  More  Data 
Handl  ing 


FIGURE  20 

PERCENT  CHANGE  IN  COMPUTER  HOURS 
DUE  TO  INDICATED  CHANGE  FROM  THE  INDEX  MEAN 
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DEVELOPMENT  ENVIRONMENT 
NOTE:**"-  Standard  Deviation 
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More  Data 
Handl  ing 


TYPE  OF  JOB 


FIGURE  21 

PERCENT  CHANGE  IN  MONTHS  ELAPSED 
DUE  TO  INDICATED  CHANGE  FROM  THE  INDEX  MEAN 
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SECTION  XX 


MACHINE-ORIENTED  LANGUAGES  (MOLs)  AND  PROCEDURE-ORIENTED  LANGUAGES  (POLs) : 
A  COMPARISON  OF  THEIR  EFFECTS  ON  THE  RESOURCE-COSTS  OF  PROGRAM  PRODUCTION 


1.  Background.  Machine-oriented  languages  (MOLs),  sometimes  called  symbolic 
or  assembly  languages,  are  unique  to  a  particular  computer  and  require  a 
processor  called  an  assembler  to  obtain  the  final  binary  machine  code.  By 
contrast,  procedure-oriented  languages  (POLs),  sometimes  referred  to  as 
compiler  or  higher-order  languages,  may  be  used  with  many  machines,  but 
require  a  computer-unique  processor  called  a  compiler  to  obtain  binary  machine 
code.  (The  distinction  is  not  always  this  clear-cut.  In  some  cases,  compilers 
produce  symbolic  statements,  which,  in  turn,  are  input  to  an  assembler.  Also, 
some  macro-assemblers  produce  more  than  one  line  of  final  machine  code  for 
each  MOL  instruction.) 

One  of  the  current  controversies  in  computer  programming  has  to  do  with  the 
pros  and  cons  of  MOLs  and  POLs  for  program  production.  Several  reasons  have 
been  advanced  for  favoring  the  use  of  POLs: 

a.  Ease  of  Coding:  Each  line  of  POL  code  written  by  a  programmer  is 
equivalent  to  several  lines  of  MOL  code,  and  hence  requires  less  work  and 
review. 


b.  Fewer  Errors:  Assuming  a  reliable  compiler,  the  fact  that  each  POL 
statement  is  equivalent  to  several  lines  of  MOL  code  (the  relationship  is 
known  as  the  POL  Expansion  Ratio)  reduces  the  opportunity  for  coding  errors. 

c.  Ease  of  Understanding:  POL  statements  are  usually  somewhat  closer  to 
natural  language.  Therefore,  POL  programs  are  more  easily  understood  and 
discussed. 

d.  Reduced  Training  for  Programming  Personnel:  The  fact  that  POL 
statements  are  easier  to  understand  means  that  junior  programmers  and  coders 
can  be  used  with  a  minimum  of  training.  (The  compilers  usually  provide 
feedback  on  many  types  of  errors  and  are  equivalent  to  additional  training. ) 

e.  Shorter  Leadtime  for  Program  Checkout:  This  is  based  on  the  reduction 
in  the  rate  of  errors  and  the  relative  ease  in  making  changes. 

f.  Reduced  Need  for  Documentation:  The  fact  that  representation  in  a 
POL  is  somewhat  closer  to  natural  language  may  eliminate  some  separately- 
written  program  design  documentation. 
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g-  Ease  of  Program  Conversion:  The  machine- independent  nature  of  POLs 
eases  the  conversion  of  programs  from  one  computer  to  another,  assuming  that 
the  necessary  compiler  is  available. 

The  MOL  advocates,  in  turn,  are  critical  of  POLs  for  several  reasons: 

a.  Obtaining  final  machine  code  through  compilers  may  consume  more 
computer  time  than  does  the  use  of  assemblers. 

b.  Final  machine  code  produced  by  compilers  tends  to  be  somevhat  longer 
(looser)  than  that  produced  by  assemblers,  for  equivalent  programming  tasks. 

c.  For  somewhat  the  same  reasons,  the  final  machine  code  produced  by 
compilers  may  take  longer  to  operate  than  does  the  code  produced  by  assemblers, 
for  equivalent  programming  tasks. 

d.  The  compiler  costs  more  time  and  money  to  develop  than  does  an  assembler. 

These  pros  and  cons  have  been  debated  and  "debunked"  by  a  variety  of  opinion- 
leaders  in  the  ADP  field(22).  Few  facts  and  fewer  numerical  data  are  available 
to  help  settle  these  controversies  (and,  in  some  cases,  the  few  data  that  have 
been  available  have,  at  times,  been  misused  to  promote  the  aims  of  a  particular 
advocate) . 

While  the  60  MOL  and  14  POL  data  points  that  are  represented  in  our  present 
data  base  are  too  few  and  some  specific  information  is  lacking  conclusively 
to  confirm  or  refute  these  claims  and  counterclaims,  we  have  made  some  pre¬ 
liminary  comparisons  in  certain  areas. 

2.  Analytical  Procedures.  The  comparisons  made  in  this  Section  were  between 
lk  programs  coded  in  a  single  POL- -JOVIAL- -and  60  programs  produced  in  a 
variety  of  MOLs  used  by  the  System  Development  Corporation. 

The  comparisons  were  made  in  terms  of  four  ratios  developed  in  the  course  of 
this  analysis. 15  Briefly,  the  ratios  indicate  the  following: 

a.  The  Program  Production  Rate  represents  the  average  net  instructions 
delivered  per  man  month  expended  on  program  design,  code,  and  test.  (This 
is  not  to  be  confused  with  the  average  amount  of  net  code  that  a  programmer 
will  deliver  in  a  month,  apart  from  the  design  and  test  phases.) 


For  these  MOL  and  POL  comparisons  only,  raw  values  of  the  ratios  were  used. 
Elsewhere  in  the  analysis,  the  values  for  the  first  three  of  these  ratios 
were  considered  after  being  logarithmically  transformed. 
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b.  The  Computer  Usage  Rate  represents  the  average  number  of  computer 
hours  expended  for  each  instruction  coded  and  delivered. 

c.  The  Documentation  Kate  represents  the  average  number  of  pages  of 
documentation  produced  for  each  instruction  coded  and  delivered. 

d.  The  POL  Expansion  Ratio  represents  the  average  number  of  words  of 
machine  code  that  are  compiled  from  each  POL  statement. 

Since  POLs,  by  their  nature,  require  fewer  statements  than  the  resulting 
number  of  machine  code  words  after  compilation  (the  POL  Expansion  Ratio 
measures  this  characteristic),  it  was  necessary  to  define  a  common  product 
in  terms  of  which  resource  expenditures  could  be  compared.  This  was  done 
by  considering  the  newly  coded  portion  of  the  delivered  sequence  of  machine 
instructions  as  the  common  standard,  produced  in  one  case  by  a  MOL  and  in 
the  other  via  a  POL.  The  definitions  of  the  four  ratios  and  their  components 
are  provided  in  Table  XHI. 

Since  we  intended  to  compare  the  means  of  the  four  ratios  for  programs  produced 
via  MOLs  and  POLs,  it  was  important  that  the  means  were  not  unduly  influenced 
by  extreme  values  (outliers).  This  is  especially  the  case  when  working  with 
very  small  samples,  such  as  the  14  POL  points.  Relying  on  well-established 
procedures ( 10 ),  we  marked  points  that  were  more  than  three  standard  deviations 
from  the  median ,  and  replaced  them  by  values  at  3.0  and  2.9  standard  deviations, 
respectively,  in  order  of  their  rank.  The  number  of  data  adjusted  in  this 
manner,  out  of  60  MOL  and  14  POL  points,  is  shown  in  Table  XIV. 

TABLE  XIV 


NUMBER  OF 

MOL  AND  POL 

POINTS  ADJUSTED 

Low 

High 

Production  Rate: 

MOL 

0 

1 

POL 

0 

0 

Computer  Usage  Rate: 

MOL 

0 

2 

POL 

0 

0 

Documentation  Rate: 

MOL 

0 

2 

POL 

0 

0 

POL  Expansion  Ratio: 

POL 

0 

0 

Since  no  macro-assemblers  were  used  in  the  production  of  the  60  MOL  programs 
represented  in  the  present  data  base,  there  is  no  MOL  expansion  ratio,  i.e., 
each  MOL  instruction  resulted  in  one  word  of  final  machine  code. 
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Considering  the  fact  that  the  MOL  and  POL  data  vere  not  ideally  suited  for 
comparisons  of  this  type,  i.e.,  the  POL  sample  was  quite  small  and  the  samples 
disparate  (l4  and  60),  some  skewness  was  apparent  for  both  the  MOL  and  POL 
data,  we  used  the  Student  "t"  statistic  as  an  appropriately  conservative 
way  17  to  compute  the  comparisons  shown  in  Table  XV. 

3.  Interpretation  of  the  Results.  Since  all  the  data  upon  which  these 
comparisons  were  based  were  taken  from  the  one  organization,  the  System 
Development  Corporation,  and  represent  one  POL,  JOVIAL,  the  results  of  the 
analysis  cannot  necessarily  be  thought  to  reflect  either  POLs  or  program¬ 
ming  generally. 

id 

On  the  other  hand,  the  substantial  difference  in  favor  of  POLs  suggests 
rather  strongly  that  they  do  require  significantly  less  manpower,  computer 
time,  and  documentation  than  would  the  same  jobs  coded  via  a  MOL. 

Unfortunately,  we  do  not  have  sufficient  data  to  test  others  of  the  claims 
and  counterclaims  that  were  summarized  earlier  in  this  Section,  e.g.,  whether 
POLs  permit  programmers  with  less  training  to  be  used,  to  what  extent  compiler 
design  can  mitigate  the  comparisons,  etc.  We  intend  to  pursue  these  and 
related  questions  as  the  data  becomes  available. 


17 

This  is  a  rather  abstract  technical  point,  but,  in  statistical  theory, 
each  piece  of  data  is  considered  as  the  mean  of  the  distribution  that  would 
result  if  every  program  were  produced  in  the  same  way  a  large  number  of 
times.  Since  we  can  say  very  little  about  the  characteristics  of  such 
distributions  (since  no  one  repeats  programming  jobs  to  collect  this  sort 
of  data),  the  Student  "t"  statistic  will  contain  terms  reflecting  this 
uncertainty.  These  terms  will  inflate  the  variation  that  enters  into  the 
statistic  and  thereby  degrade  the  confidence  we  have  in  our  inferences. 

In  fact,  then,  the  significances  of  the  differences  between  the  ratio  means 
as  shown  in  Table  XIV  are  quite  conservative,  i.e.,  the  probability  of 
the  differences  occurring  by  chance  may  be  much  less,  but  we  do  not  have  the 
data  to  demonstrate  this  case. 

18 

Some  experts  say  that  the  JOVIAL  compiler  produces  programs  that  are  10 
to  15  percent  longer,  in  final  machine  code,  than  if  they  had  been  written 
in  a  MOL.  While  we  did  not  correct  for  this,  the  magnitude  of  the  ratio 
differences  is  large  enough  that  the  implications  would  not  have  changed 
significantly. 
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(Page  98  Blank) 


APPENDIX  I 


THE  DATA  COLLECTION  QUESTIONNAIRE 


The  data  from  74  computer  programs  in  the  final  base  for  this  analysis  were 
collected  in  one  of  two  ways.  For  51  programs,  the  questionnaire  shown  in 
this  Appendix  was  completed.  The  remaining  23  programs  were  carried  over 
from  the  previous  analysis  and  questionnaire  (TM-l447/00l/00) ,  and  required 
the  use  of  a  supplementary  questionnaire  to  make  the  two  portions  of  data 
base  equivalent.  The  questions  that  constituted  the  supplement  are  marked 
by  a  black  square,  I,  in  this  Appendix. 
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GENERAL  INSTRUCTIONS 


This  questionnaire  is  a  means  for  collecting  data  on  completed  programming  efforts.  These 
data  will  help  us  to  identify  and  verify  key  factors  affecting  the  cost  of  computer  programs. 
We  are  seeking  to  increase  the  reliability  of  techniques  for  estimating  costs  of  future 
program  development . 

The  questionnaire  is  organized  into  seven  parts  so  that  each  part  may  be  easily  delegated. 

The  first  part  summarizes  the  major  costs  of  the  program  being  examined  in  terms  of  man 
months,  computer  hours,  and  calendar  time  involved.  The  six  remaining  parts  are  questions 
concerning  factors  that  may  affect  the  cost  of  computer  programs  and  are  organized  as 
follows : 


A.  Operational  Requirements  and  Design 

B.  Program  Design  and  Production 

C.  Data  Processing  Equipment 

D.  Programming  Personnel 

E.  Management  Procedures 

F.  Development  Environment 

Generally  speaking,  the  first  two  categories  address  the  question,  "What  was  the  job  to  be 
done?"  The  next  two  ask,  "What  were  the  available  resources?"  and  the  last  two  examine, 

"What  was  the  nature  of  the  working  environment?" 

The  information  we  are  seeking  is  fairly  detailed  and  will  require  some  effort  to  compile. 

To  minimize  this  effort,  we  have  attempted  to  make  the  questions  as  clear  and  definitive  as 
possible.  Some  of  the  questions  may  be  answered  readily,  whereas  others  required  that  we 
develop  definitions  before  these  questions  could  be  understood.  We  encourage  answering 
all  questions  even  if  a  best  guess  or  estimate  is  all  that  is  available.  When  there  is 
some  question  about  the  accuracy  of  a  numerical  answer,  give  the  range  of  values  about 
which  the  midpoint  is  the  most  probable  value. 

In  order  that  the  programs  we  sample  be  homogeneous  in  at  least  one  dimension,  we  have  defined 
the  bounds  on  each  program  to  be  examined.  A  program  satisfying  the  following  definition  is 
termed  a  program  data  point. 

A  program  data  point  is  the  smallest  set  of  computer  program  instructions 

(1)  whose  purpose  is  defined  by  someone  other  than  the  programmer, 

(2)  that  is  delivered  to  the  user  (customer)  as  a  package,  and 

(3)  that  is  loaded  into  the  computer  as  a  program  unit  or  system  to  achieve  the  stated 
purpose  or  objective. 

One  complete  questionnaire  is  required  for  each  program  corresponding  to  a  data  point.  We 
will  need  your  help  in  identifying  data  points  in  accordance  with  the  above  definition. 

By  this  definition,  a  program  data  point  can  be  an  operational  program,  a  utility  program, 
or  even  an  experimental  program.  These  are  clearly  not  limited  to  any  specific  function. 
Similarly,  the  user  of  the  program  (represented  by  the  data  point)  may  be  the  buyer,  but 
he  may  also  be  another  programmer,  as  in  the  case  of  a  utility  program.  The  responder 
must  keep  in  mind  at  all  times  the  portion  of  the  program  system  that  he  is  calling  the 
program  data  point  when  answering  the  questions.  Additionally,  the  definition  of  a  program 
data  point  includes  the  following  limits  on  the  scope  of  activities  considered  to  be  the 
programming  process.  Here,  we  are  concerned  with  the  activities  of  program  design,  code, 
test,  and  documentation. 

Finally,  we  encourage  you  to  note  any  qualifying  remarks,  or  discussion  concerning  your 
answers  on  the  reverse  side  of  the  answer  sheet.  Similarly,  we  would  appreciate  any 
comments  or  suggestions  concerning  the  nature  of  the  questions  themselves. 
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SUMMARY  OF  COSTS 

1.  Total  number  of  man  months*  to  program  design,  code, 
test,  and  document  this  program,  including  first-line 
supervision.  Do  not  include  the  cost  of  any  associated 
executive  or  utility  program. 

2.  Number  of  man  months  to  develop  utility  programs. 

3.  Maximum  number  of  programmers  employed  on  this  program. 

4.  Number  of  months  that  more  than  90$  of  the  maximum 
number  of  programmers  were  employed. 

5.  Start  date  for  program  design.  _ 

Month 

By  program  design,  we  mean  the  activity  whose  inputs  are  the 
operating  system  description  and  operational  specifications 
and  whose  outputs  are  program  design  specifications  and  flow 
charts. 


6.  Completion  date  for  program  delivery. 

Month 

By  program  delivery,  we  mean  the  point  at  which  the  program 
is  ready  to  be  installed  in  the  operational  computer  and 
begin  its  system  test  in  the  environment  for  which  the  pro¬ 
gram  was  designed. 


7.  Total  number  of  computer  hours  used. 

Type 

8.  a.  Number  of  man  trips  for  briefings,  problem 

solving,  concurrence,  etc. 

b.  What  was  the  average  round-trip  distance/trip? 


Year 


Year 


Hours 


♦Indicate  effective  man  months  for  analysts,  programmers,  and  support 
personnel. 


APPENDIX  I  (Cont ’d) 


A.  OPERATIONAL  REQUIREMENTS  AND  DESIGN* 

1.  Was  there  a  requirement  for  innovation  in 
the  information  processing  system? 


Check  Appropriately 


Yes  No 


By  Innovation,  ve  mean  either  a  new  data-processing  applica¬ 
tion  of  a  known  programming  technique  and/or  a  new  technique 
for  a  known  application.  By  new,  we  mean  new  to  the  people 
involved. 


2.  Was  there  participation  by  the  programming 
organization  in  the  requirements  analysis 
and/or  operational  design?  Yes  _  No 

The  requirements  analysis  is  conducted  to  specify  in  detail 
the  performance  requirements  of  this  information-processing 
system.  These  performance  requirements  axe  the  input  to  the 
operational  design  activity,  which  translates  the  require¬ 
ments  into  operational  design  specifications.  These  speci¬ 
fications  indicate  how  the  information-processing  needs  will 
be  satisfied. 


3.  How  well  were  the  operational  requirements 

known  and  documented?  In  great  detail 

In  broad  outline 
Only  vaguely 

4.  How  many  organizational  users  supply 
inputs  and  request  outputs  from  this 
program? 

5.  How  many  automatic  data-processing  centers 
are  in  the  system? 


Here  we  mean  computer-based  centers  with  automatic  digital 
communications . 


*Part  A  pertains  to  the  information* processing  system  in  which  the 
computer  system  is  embedded.  Part  B  is  concerned  with  program 
design  considerations. 
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■  6.  Characterize  the  complexity  of  the  overall 
information-processing  system  on  a  scale  of 
(l)-(5)  from  simple  to  highly  complex. 

While  almost  1  of  the  previous  questions  in  this 
section  relate  to  the  complexity  of  the  information¬ 
processing  system,  this  question  attempts  to  aggregate 
these  factors  into  one  index  of  complexity.  Select 
that  descriptor  that  most  nearly  applies. 


Complexity  Scale 

(1)  Simple  operational  functions  and  few  users.  Direct  translation  of  manual 
tasks  to  automatic  functions.  Off-line  output  with  no  interfacing  agencies. 

(2)  Operational  requirements  fairly  well  understood.  Only  a  few  new  tasks  to  be 
automated.  No  elements  of  on-line  or  real-time  operations. 

(3)  Operational  requirements  known  in  only  a  broad  outline.  Operational  functions 
to  be  performed  require  some  analysis  to  determine  how  best  to  design 
information-processing  system.  System  may  or  may  not  be  real  time,  has  few 
interfacing  agencies  providing  input  and  requiring  output. 

(4)  Operational  requirements  known  only  vaguely.  Many  new  functions  requiring 
decision-making  capabilities.  Mixture  of  on-line  and  off-line  responses  to 
many  diverse  users. 

(5)  Many  new  operational  functions,  hardware  components  to  be  developed  and  inter¬ 
acting  users.  Operational  requirements  highly  unstructured  and  indeterminate. 
System  must  have  great  capability  for  reacting  to  new  situations  on  real  time 
basis. 
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B.  PROGRAM  DESIGN  AND  PRODUCTION 

■  1.  Indicate  the  number  of  computer  program 

instructions  and  words.  EXCLUDE  the  contents 
of  the  data  base  and  tables  used  for  storage 
of  input  and  output  messages  and  interprogram 
communications . 

By  instructions,  we  mean  machine  language  instructions  or 
orders^  If  the  computer  is  a  multiaddress  machine,  count 
each  instruction  separately.  If  a  procedure-oriented 
language  (POL)  is  used,  list  the  number  of  POL  statements 
in  parenthesis  alongside  the  resulting  machine  instructions. 

By  subprogram,  we  mean  the  first  level  in  the  logical  sub¬ 
division  of  data-processing  functions  in  the  program  being 
considered. 

By  subroutine,  we  mean  some  well  defined,  logical  or 
mathematical  function. 


■  a.  Total  instructions  in  delivered  program. 

.  New  instructions  written  for  this  program 
system:  in  subroutines,  logical  blocks, 
subprograms  inserted  in  programmed  portions. 

.  Reused  instructions:  in  subroutines, 
logical  blocks,  and  subprograms  obtained 
from  previous  versions  of  this  program 
or  libraries. 

■  b.  Total  instructions  written  but  discarded  and 

not  delivered. 

.  Number  of  instructions  discarded  due  to 
operational  changes. 

.  Number  of  instructions  discarded  due  to 
error  corrections. 

c.  Number  of  subprograms  in  this  program  system. 

2.  Total  number  of  logical  words  (items)  in  the  data 
base. 


By  data  base,  we  mean  the  subset  of  tables  that  describe 
the  environment  of  the  problem  that  the  program  is  solving 
and/or  the  files  to  be  processed.  Here,  if  the  data  base 
changes  in  size  relatively  often,  indicate  an  average  size. 
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3.  Total  number  of  classes  of  items  in  the  data  base. 


By  classes,  we  mean  categories  or  types  of  items  such  as 
names  of  people,  salaries,  cities,  states,  or  any  charac¬ 
teristics  of  information  for  which  there  are  many  items  or 
entries. 


4.  Total  number  of  words  in  tables  and  constants  not 
included  in  the  data  base. 


5.  Indicate  the  number  of  message  types  handled  by 

the  program  by:  Input  message  types 

Output  message  types 


By  message  type,  we  are  not  concerned  here  with  the  equip¬ 
ment  involved,  but  are  interested  in  the  nature  of  the  input 
message,  such  as  flight  plan,  position  report  or  sensor  data, 
and  output  message  types  such  as  displays  and  reports. 


■  6.  Characterize  the  complexity  of  the  program  design  by  the 
scale  (l)  to  (5).  Select  the  descriptor  that  most  nearly 
applies. 


Complexity  Scale 

(1)  Initial  program  design  carried  through  without  change;  subprograms  are  nearly 
independent,  like  closed  subroutines;  intercommunications  between  programs  at 
a  minimum;  no  storage  and  timing  problem. 

(2)  Few  changes  to  initial  program  design;  programmers  familiar  with  this  type  of 
design;  interprogram  communication  low,  no  need  for  common  data  pool;  less 
than  10$  reprogrammed  due  to  nonoperational  changes. 

(3)  Frequent  changes  in  program  design;  reassignments  of  personnel  to  assist  in 
areas  estimated  too  low;  over  10$  of  effort  going  to  solve  program  inter- 
ccranunication  problems  and  reprogramming;  data  pool  considered  but  need 
marginal.,  might  have  been  useful  in  checkout. 

(h)  Many  changes  to  initial  program  design;  personnel  unfamiliar  with  many 

concepts,  up  to  50$  of  effort  needed  for  interprogram  communications,  data 
pool  needed  or  should  have  been  used;  over  50$  reprogrammed  to  solve  non¬ 
operational  problems,  such  as  table  redesign,  errors  found  in  checkout,  etc. 

(5)  Initial  program  design  completely  remade;  over  50$  of  personnel  reassigned, 
well  over  half  of  effort  goes  to  communications  problems;  storage  and  timing 
problems  of  the  subprograms  affect  several  other  subprograms;  discarded 
registers  approach  the  size  of  the  final  program. 
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■  7*  Characterize  the  nature  of  the  data  processing  by 

indicating  the  percentage  of  the  program's  instructions 
devoted  to  the  following: 


.  Clerical 
.  Mathematical 
.  Input/ Output 


Logical-Control 
Self-checking,  FIX 
Other  (specify) 


Clerical 
Mathematical 
Input/ Output 
Logical-Control 

Self-Checking,  FIX 


Examples 

-  bookkeeping,  sorting,  searching,  file  maintenance. 

-  evaluate  formula  for  given  parameters,  solve  equations. 

-  format  data  for  output  equipment,  translate  input  messages. 

-  sequence  the  data-processing  operations  and  i/o  according  to 
orders,  priorities  or  timing  requirements. 

-  monitoring  programs  which  detect  and  report  errors;  some  try 
to  repair  the  effects  of  generated  errors. 


■  8.  Characterize  the  program  function  as  percentages  of 
one  or  more  of  the  following  major  functions: 

.  information  storage  and  retrieval 

.  data  acquisition  and  display 

.  control  or  regulation 

.  decision  making;  choosing  an  optimum 

.  transformation;  reformatting  data 

.  generation  to  produce  desired  outputs 

.  other  (specify) 

■  9*  What  is  the  average  operate  time  of  the  completed 

program? 

■  10.  How  frequently  does  the  completed  program  cycle  or 

operate?  (e.g.,  number  of  times/day  or  week) 
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11.  Indicate  the  existence  of  constraints  on  program 
design,  such  as 


insufficient  memory  capacity 

Yes 

No 

insufficient  input/output  capability 

Yes 

No 

stringent  timing  requirements 

Yes 

No 

other  (specify) 

Yes 

No 

12.  What  language  was  used  in  coding  the  program? 


■  13.  The  following  concerns  utility  and  support 
programs  used  in  developing  this  program: 

These  are  programs  for  generation,  simulation  and 
reduction  of  data;  control  and  facility  programs 
for  modification,  error  correction,  intervention 
and  error  detection  and  program  for  diagnostics, 
loading,  editing  and  file  maintenance  of  the 
system  being  designed. 

■  a.  Number  of  instructions  in  required  support 
programs . 


■  b.  Number  of  distinct  support  programs  used. 


■  c.  Number  of  support  programs  available  without 
cost  at  the  beginning  of  the  job. 


■  d.  Number  of  instructions  in  the  support  programs 
available  without  cost. 


14.  Were  there  documented  test  plans  and  test  designs 
which  specified  input  test  data  and  expected 

outputs?  Yes  _  No _ 

15.  What  was  the  extent  of  documentation?  ^  , 

Internal  External 

a.  Number  of  different  types  of  documents. 


b.  Number  of  pages  in  one  set  of  documents. 


By  types  of  documents,  we  mean  documents  such  as  Operating 
System  Description,  Program  Design  Specifications,  Status 
Reports,  Error  Correction  Reports,  Program  Listings,  etc. 

By  internal,  we  mean  for  the  programming  organization's 
use. 


By  external,  we  mean  for  delivery  to  the  customer. 
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C.  DATA  PROCESSING  EQUIPMENT 

1.  a.  Give  name(s)  of  the  computer(s)  vised  in 
the  development  of  this  program  and  size 
of  core  storage. 


I  b.  Was  this  your  first  program  development 

effort  on  this  computer?  Yes  No 

H  2.  a.  On  the  average,  what  was  the  turnaround 
time  experienced  by  the  programmers? 


By  turnaround  time,  we  mean  the  total  elapsed  time  between 
submission  and  return  of  a  computer  run. 

3*  How  many  automatic  data-processing  components 
were  developed  concurrently  with  this  program? 


By  ADP  components,  we  mean  those  pieces  of  equipment  that 
are  recognized,  addressed,  or  controlled  by  the  computer 
program  (e.g.,  display  consoles,  message  composers,  AD 
converters ,  etc . ) . 


4.  a.  List  the  number  and  types  of  (CRT)  display 
equipment  driven  by  this  program. 


b. 


List  the  number  and  types  of  on-line  input/output 
equipment  (e.g.,  tapes,  typewriters,  etc.). 


108 


APPENDIX  I  (Cont'd) 


D.  PROGRAMMING  PERSONNEL 

1.  Fill  in  the  following  table  with  average* 
values : 

.  number  of  programmers  by  type 

.  years  of  experience  with  language  used 

.  years  of  experience  with  computer  used 

.  years  of  experience  with  this 
application 


I  II  III  IV 


Type  Position 


Description 


I  Coder  Writes  machine  language  instructions  from  flow  charts.  Helps 

prepare  flow  charts  and  test  programs. 


II  Programmer  Develops  programs  to  solve  well-defined  problems.  Prepares 
flow  charts,  writes  instructions,  tests  programs,  modifies 
established  computer  programs. 


Ill  Senior 

Programmer 


Conceives,  develops  and  improves  large,  complex  computer 
programs,  e.g.,  automatic  programming  routines.  Improves 
efficiency  of  existing  programs. 


IV  System  Formulates  and  plans  new  program  system  applications.  Keeps 

Programmer  abreast  of  related  economic  disciplines  and  new  information 

processing  technology.  Is  highly  creative  in  designing  and 
developing  major  computer  program  systems. 


2.  a.  How  many  programmers  participated  in  the 
program  design? 

b.  How  many  programmers  worked  on  the  program 
for  the  entire  duration  of  the  project? 

■  c.  What  was  the  average  programmer  turnover  rate? 


By  turnover,  we  mean  the  number  of  programmers  replaced 
per  month  divided  by  the  total  programmer  staff. 


*By  average,  we  mean  the  mode,  or  that  number  that  occurs  most  often. 
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E.  MANAGEMENT  PROCEDURES 

1.  Was  there  a  documented  plan  or  procedure  for  the 

following? 

a.  Evaluation  and  implementation  of  system 
design  changes. 

b.  Evaluation  and  implementation  of  program 
design  changes. 

c.  Dissemination  of  error-detection  and 
error-correction  information. 

d.  Use  of  the  computer  facility,  e.g., 
allocation  of  time,  number  of  runs  per 
day  per  programmer. 

e.  Contingency  plan  in  the  event  the  computer 
was  overloaded  or  otherwise  unavailable. 

f.  Communicating  with  other  agencies. 

g.  Concurrence  on  design  specifications. 

h.  Cost  control. 

i.  Management  control  in  the  form  of  PERT 
or  Gantt  charts. 

«i.  Document  control  (e.g.,  design  file  and 
documentation  standards). 

k.  Standards  for  coding,  flow  charts,  etc. 
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DEVELOPMENT  ENVIRONMENT 


1.  a.  State  the  number  of  agencies  whose 

concurrence  was  required  on  operational 
design  specifications. 


By  concurrence  here,  we  mean  the  actual  documented  agree- 
ment  and  approval  that  the  specifications  are  understood 
and  satisfactory  to  the  agency  concerned.  The  activity  of 
concurrence  consists  of  preparation  for  briefings,  delivery 
of  briefings  and  meetings  intended  to  discuss  the  content 
of  the  design  specifications. 


■  b. 


Estimate  customer  experience  and  knowledge 
concerning  the  development  of  automatic  data 
processing  systems. 


Extensive 


Limited 


None 

2.  Was  the  computer  operated  by  an  agency  other 
than  the  program  developer? 


3.  Was  the  computer  facility  operated  on  the 
basis  of: 


Yes  _  No 

Open  shop 


Closed  shop 


Time-sharing 


4.  a.  Was  the  program  developed  at  a  site 
other  than  the  operational  location? 

b.  Is  the  computer  at  the  operational  site 
different  than  the  computer  used  for 
program  development? 


Yes 


Yes 


No 


No 


5.  Did  the  program  development  take  place  at 
more  than  one  location  during  the  effort? 


Yes 


No 
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INDEX  AND  CODING  OF  VARIABLES 


In  this  Appendix,  each  variable  used  in  the  analysis  is  identified  in  at 
least  two  of  three  ways : 

o  by  the  pert  and  question  number,  if  it  corresponds  to  a  specific 
item  in  the  questionnaire  (Appendix  i). 

o  by  a  number  that  is  associated  with  each  variable  throughout  the 
analysis  (whether  or  not  the  variable  is  taken  directly  from  the 
questionnaire  or  formed  in  some  way,  e.g.,  a  ratio). 

o  by  a  short  description  of  the  variable  (and  its  formation,  where 
applicable). 

The  coding  of  each  variable  is  also  indicated. 

The  question  numbers  in  the  first  column  correspond  to  the  seven  parts  of 
the  questionnaire. 

S  -  Summary  of  Major  Costs 
A  -  Operational  Requirement  and  Design 
B  -  Program  Design  and  Production 
C  -  Data  Processing  Equipment 
D  -  Programming  Personnel 
E  -  Management  Procedures 
F  -  Development  Environment 
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APPENDIX  III 


A  PARTIAL  DATA  BASE  MATRIX 


The  data  contained  in  this  Appendix  are  a  partial  tabulation  of  responses  to 
the  data  collection  questionnaire  and  represent  the  variables  that  were  con¬ 
sidered  in  the  formal  analysis.  The  data  presented  may  not  be  in  the  form 
that  they  appear  in  the  questionnaire  for  the  following  reasons: 

a.  Transformation  to  ratios,  e.g.: 

Documentation  Rate  =  Pages  Documented 

Net  Instructions  Coded 

b.  Transformation  to  percentages,  e.g.: 

%  New  Instructions  =  Number  of  New  Instructions 

Total  Delivered  Instructions 

c.  Transformation  to  logarithms,  e.g.: 

Log  Man  Months  =  Log10  (Total  Man  Months) 

d.  Formation  of  variable  from  two  or  more  variables,  e.g.: 

Months  Elapsed  =  Completion  Date  -  Start  Date 

The  following  conventions  were  carried  through  from  the  computer  output  to 
this  Appendix: 

a.  The  first  60  sample  points  represent  programs  coded  in  machine- 
oriented  language;  the  last  14  represent  those  coded  in  either 
procedure -oriented  language,  or  both  machine-  and  procedure-oriented 
languages. 

b.  The  logarithm  of  zero  is  represented  by  the  number  zero;  the  logarithm 
of  one  is  represented  by  a  negative  zero,  e.g.,  (-0). 

c.  A  negative  entry  for  a  logarithm  represents  the  result  of  subtracting 
the  ten's  complement  of  the  corresponding  number  from  ten,  e.g. 

i.e.,  Log10  0.1400  =  9.1461  -  10.0000  =  -  .8539 

For  a  complete  identification  and  definition  of  variables,  the  reader  is 
referred  to  Appendix  II  of  this  Report. 
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APPENDIX  IV 


VALIDITY  OF  SELECTED  PREDICTORS 


This  Appendix  contains  the  correlations,  (i.e.,  validities),  of  selected 
predictors  with  the  major  dependent  variables.  The  dependent  variables  are 
identified  in  column  headings;  the  predictors  are  noted  both  by  title  and 
by  the  identifying  number  used  throughout  the  analysis. 

A  more  detailed  identification  of  all  variables  is  contained  in  Appendix  II. 
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APPENDIX  V 


A  LIST  OF  MAJOR  DEPENDENT  AND  INDEPENDENT  VARIABLES 


This  Appendix  lists  the  dependent  and  independent  variables,  and  their  identi¬ 
fying  numbers,  that  remained  after  preliminary  statistical  analysis  to  remove 
redundant  and  spurious  relationships.  For  example,  certain  variables  were 
eliminated  from  further  consideration  after  correlation  analysis  indicated 
that  they  were  weakly  related  to  resource-costs  and  would  contribute  little 
to  prediction. 
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APPENDIX  V  (Cont’d) 


ID  No.  Tltle 

10  Log^o  number  of  trips 

11  Logio  average  distance/trip 

12  Innovation  in  system 

13  Participation  by  programming  organization  in  requirement  and/ or 

operational  design 

14  How  well  were  operational  requirements  known  and  documented 

15  Log^o  number  of  users  supplying  input  to  and  requiring  output 

from  program 

16  Logio  nuraber  of  A®15  centers  in  system 

17  Characterize  complexity  of  overall  system 

21  Logio  number  of  instructions  written  for  this  program  (POL) 

30  Logio  number  of  subprograms 

31  Logio  number  of  words  in  data  base 

32  Logio  numl:,er  of  classes  of  items  in  data  base 

33  Logio  number  of  wol>ds  in  tables  and  constants  not  in  data  base 

34  Logic  number  of  input  message  types 

35  Logic  number  of  output  message  types 

3 6  Characterize  complexity  of  program  design 

37  Percent  clerical  instructions 

38  Percent  math  instructions 

39  Percent  i/o  instructions 

40  Percent  logical  control  instructions 

41  Percent  self-checking-fix  instructions 

42  Percent  information  storage  and  retrieval 

43  Percent  data  acquisition  and  display 

45  Percent  decision  making 

46  Percent  transformation- reformatting  data 

47  Percent  generation  to  produce  desired  output 

50  Insufficient  manory  capacity 

51  Insufficient  i/o  capacity 

52  Stringent  timing  requirements 


158 


APPENDIX  V  (Cont'd) 


ID.  No. 


Title 


64 

65 

66 
87 
99 

100 

101 

102 

103 

104 

105 

106 

116 

117 

118 
122 

125 

126 
127 
130 
132 
134 
138 

141 

142 

143 

144 
1 
9 

20 


First  programming  effort  on  computer 

i'C&LO  average  turnaround  time  experienced  by  programmers 
Log10  number  of  ADp  components  developed  with  program 
L°s10  average  turnover  rate 

Number  of  agencies  whose  concurrence  was  required  on 
operational  design 

Estimated  customer  experience 

Was  computer  operated  by  agency  other  than  program  developer 
Open/closed  shop 
Time  sharing 

Program  developed  away  from  operational  location 

Computer  at  operation  site  different  than  at  development  site 

Program  developed  at  more  than  one  location 

Log^Q  miles  traveled 

Logic  travel  rate 

L°®10  Pa®es  documentation 

Log10  staffing  stability  index 

Logic  throughput  index 

Logic  data  retrieval  index 

Log10  data  turnover  index 

Percent  new  instructions  (machine) 

Percent  reused  instructions  (machine) 

Management  index 
Log^c  senior  man  months 

Log10  average  senior  programmer  experience 

Logic  average  junior  programmer  experience 

Percent  programmer  participating  in  design 

Percent  programmers  for  duration  of  project 

Log^o  total  man  months 

Logic  total  computer  hours 

Log10  new  instructions  written  (machine) 
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APPENDIX  V  (Cont’d) 


ID.  No. 

Title 

54 

L°®10  num^er  instructions  in  support  programs 

109 

Percent  error  rate  (machine) 

111 

Log^Q  production  rate  (machine) 

115 

Percent  operational  discards 

119 

Logio  documentation  rate  (machine) 

121 

Log10  months  elapsed 

123 

Log^  computer  usage  rate  (machine) 

128 

Percent  senior  programmers 

i4o 

Log^Q  average  programmer  experience 

113 

Logio  percent  utility  instructions  (machine) 
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APPENDIX  VI 


A  LIST  OF  MAJOR  INDEPENDENT  VARIABLES 


The  36  predictor  variables  listed  in  this  Appendix  were  obtained  from  the  69 
dependent  and  independent  variables  listed  in  Appendix  V  by  eliminating  the 
primary  dependent  variables,  and  other  variables  that  also  represented 
resource-costs,  e.g.,  Number  of  Trips. 
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ID.  I 

12 

17 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

4o 

4l 

42 

47 

50 

51 

52 

64 

65 

66 

100 

101 

102 

103 

104 

105 

106 

22 


APPENDIX  VI  (Cont'd) 


Title 

Innovation  in  system 

Characterize  complexity  of  overall  system 

Logic  number  of  subprograms 

Log10  number  of  words  in  data  base 

Logfc  number  of  classes  of  items  in  data  base 

Log10  number  of  words  in  tables  and  constants  not  in  data  base 

Logic  number  of  input  message  types 

Log^c  number  of  output  message  types 

Characterize  complexity  of  program  design 

Percent  clerical  instructions 

Percent  math  instructions 

Percent  i/o  instructions 

Percent  logical  control  instructions 

Percent  self-checking- fix  instructions 

Percent  information  storage  and  retrieval 

Percent  generation  to  produce  desired  output 

Insufficient  memory  capacity 

Insufficient  i/o  capacity 

Stringent  timing  requirements 

First  programming  effort  on  computer 

Logio  average  turnaround  time  experienced  by  programmers 
Log^0  number  of  ADP  components  developed  with  program 
Estimated  customer  experience 

Was  computer  operated  by  agency  other  than  developer 
Open/closed  shop 
Time  sharing 

Program  developed  away  from  operational  location 

Computer  at  operational  site  different  than  at  development  site 

Program  developed  at  more  than  one  location 

Log^0  number  of  reused  instructions  (machine) 

Management  index 

Percent  programmers  participating  in  design 
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APPENDIX  VI  (Cont’d) 


ID.  No.  Title 

109  Percent  error  rate  (machine) 

111  Log^o  production  rate  (machine) 

115  Percent  operational  discards 

128  Percent  senior  programmers 
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APPENDIX  VII 


FACTOR  LOADINGS  OF  SELECTED  VARIABLES  AND 
DERIVATION  OF  THE  COSTLINESS  FACTOR 


This  Appendix  summarizes  the  results  of  the  factor  analysis  of  69  dependent 
and  independent  variables,  which  was  used  to  obtain  the  Costliness  Factor. 

The  variables  considered  in  the  factor  analysis  are  given,  as  well  as  the 
loadings  for  the  first  factor .  Also,  the  statistical  relationships  resulting 
from  the  regression  of  Costliness  Factor  against  its  components,  to  obtain 
proper  weighting,  are  shown. 
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APPENDIX  VII  (Cont'd) 


).  No. 

Title 

Factor 

Loadings 

10 

1,0^0  number  of  trips 

73 

11 

Logic  average  distance/trip 

51 

12 

Innovation  in  system 

32 

13 

Participation  by  programming  organization 

20 

l4 

How  well  were  plans  known  and  documented 

-5 

15 

Log^  number  of  users  supplying  input 

23 

16 

Log^  number  of  ADP  centers 

-20 

17 

System  complexity 

54 

19 

Log^  new  instructions  written  (POL) 

-2 

30 

Log10  number  of  subprograms 

64 

31 

Log10  number  of  words  in  data  base 

47 

32 

Log^Q  number  of  classes  in  data  base 

32 

33 

Log^Q  number  of  words  not  in  data  base 

4o 

34 

Logic  Number  of  input  message  types 

4i 

35 

Log10  number  of  output  message  types 

38 

36 

Complexity  of  program  design 

59 

37 

Percent  clerical  instructions 

-35 

38 

Percent  math  instructions 

21 

39 

Percent  i/o  instructions 

4 

40 

Percent  logical  control 

17 

4i 

Percent  self-check- fix 

4 

42 

Percent  information,  storage  and  retrieval 

-19 

43 

Percent  data  acquisition  and  display 

15 

45 

Percent  decision  making 

2 

46 

Percent  transform-reformatting 

-22 

47 

Percent  generation 

37 

50 

Insufficient  memory  capacity 

36 

51 

Insufficient  i/o  capacity 

28 

52 

Stringent  timing  requirements 

31 

64 

First  programming  effort  on  computer 

27 
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ID.  ] 

65 

66 

87 

99 

100 

101 

102 

103 

104 

105 

106 

116 

117 

118 

122 

125 

126 

127 

130 

132 

134 

138 

l4l 

142 

143 

144 


APPENDIX  VII  (Cont'd) 


Title 


Factor 

Loadings 


Log^0  average  turnaround  time 

Log10  number  ADP  components  developed  with  program 
Log^0  average  turnover  rate 
Number  of  agencies  concurring  in  design 
Estimate  customer  experience 

Was  computer  operated  by  agency  other  than  developer 
Open/close  shop 
Time  sharing 

Program  developed  away  from  operational  location 

Operational  computer  different  than  development  computer 

Program  developed  at  more  than  one  location 

Log^0  miles  traveled 

Log^  travel  rate 

Log^  pages  documentation 

Log^  staffing  stability  index 

Logic  throughput  index 

Logic  data  retrieval  index 

Logio  data  turnover  index 

Percent  new  instructions  (machine) 

Percent  revised  instructions  (machine) 

Management  Index 
LogiQ  senior  man  months 

Log10  average  senior  programmer  experience 
Logic  average  junior  programmer  experience 
Percent  programmers  participating  in  design 
Percent  programmers  for  duration  of  project 
Logic  total  man  months 
Log^c  total  computer  hours 
Log^c  new  instructions  (machine) 


13 

33 

-40 

18 

-17 


-1 


k 

-23 

17 

5 

26 

67 

3 

80 

26 

79 

-9 

29 

17 

-17 

38 

81 

-34 

-68 

-46 

-28 

91 

88 

88 
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APPENDIX  VII  (Cont'd) 


ID.  No. 

Title 

Factor 

Loadings 

54 

Logic  number  support  instructions 

39 

109 

Percent  error  rate  (machine 

6 

ill 

Log10  Production  (machine) 

13 

115 

Percent  operational  discards 

23 

119 

Logio  documentation  rate  (machine) 

-19 

121 

Logio  months  elapsed 

77 

123 

Logio  computer  usage  rate  (machine 

-5 

128 

Percent  senior  programmers 

-14 

l4i 

Logic  average  programmer  experience 

-68 

1^5 

Dog^c  percent  utility  instructions  (machine) 

-4o 
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APPENDIX  VII  (Cont'd) 

CORRELATION  AND  REGRESSION  ANALYSIS: 
COSTLINESS  FACTOR  (15*0  >  COSTLINESS  FACTOR  COMPONENTS 


Log10  No* 

Log10  Pgs. 

L°6l0  Total 

Logic  Total 

Log^ ,  Net. 

of  Trips 

Document . 

Man  Months 

Comp.  Hrs. 

Instr. 

Intercorrelations  of  Unweighted  Components 

10 

0 

*oH 

0 

number  of  trips 

1.00 

.51 

.62 

•  6l 

•  55 

118 

LOg10 

pages  documented 

.51 

1.00 

.81 

.72 

.74 

1 

Lo&lo 

total  man  months 

.62 

.81 

1.00 

.86 

.87 

9 

L°glO 

total  computer  hours 

.61 

.72 

.86 

1.00 

.66 

20 

LOg10 

new  instructions  (machine) 

.55 

.74 

•  87 

.86 

1.00 

Table  of  Validity  and  Correlation  Coefficient 

Validity  of  adjusted  components 
with  costliness  factor 

.73 

.80 

•  91 

.88 

.88 

Adjusted  "I 

>"  weights 

6.23 

3.34 

8.44 

b.bQ 

6.34 

Adjusted  Multiple  Correlation  Coefficient 

.96 


Adjusted  Standard  Error  of  Estimate 

5.74 


Adjusted  Prediction  Equation 

Y154  -  6.23^  +  3.3^8  +  8.44^  4.48x9  +  6.^^  -  58.63 
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APPENDIX  VIII 


CORRELATION  AND  REGRESSION  ANALYSIS:  THE  DEVELOPMENT 
OF  EQUATIONS  FOR  ESTIMATING  THE  MAJOR  DEPENDENT 
VARIABLES  USING  SELECTED  PREDICTORS 


This  Appendix  provides  the  details  of  four  estimating  equations  that  use  six 
to  eight  individual  variables  as  predictors,  instead  of  task  indices.  The 
equations  are  comparable,  in  this  sense,  to  those  of  the  previous  analysis  of 
this  series  (TM-l447/00l/00),  expect  that  Number  of  New  Instructions  is  not 
used  as  a  predictor. 
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APPENDIX  IX 


CORRELATION  AND  REGRESSION  ANALYSIS: 

THE  DERIVATION  OF  WEIGHTS  FOR  THE  COMPONENTS  OF  THE  UNIQUENESS  INDEX 


This  Appendix  illustrates  the  procedures  that  were  used  to  obtain  weights  for 
the  components  of  each  of  the  task  indices.  The  Uniqueness  Index,  variable 
1^9,  is  used  as  an  example. 

The  five  major  dependent  variables  were  first  regressed  against  the  components 
of  the  Uniqueness  Index.  The  details  are  shown  in  this  Appendix  for  each  case. 
Next,  the  regression  coefficients  were  averaged  and  rescaled  to  obtain  final 
weights.  This  process  is  also  detailed. 
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VARIABLE  1,  LOG10  TOTAL  MAN  MONTHS 
AGAINST  COMPONENTS  OF  UNIQUENESS  INDEX 


First  Prog.  Dev. 

Stringent  Effort  on  at  More 

Innovation  Timing  Computer  Than  1  Loc . 


Intercorrelation  of  Unweighted  Uniqueness 

Components 

12  Innovation  in  System 

52  Stringent  Timing  Requirements 

64  First  Programming  Effort  on  Computer 

106  Program  Developed  at  More  Than  One 
Location 

Table  of  Validity  and  Regression  Coefficients 

Validity  of  Unweighted  Uniqueness  Components 
with  I*>610  Total  Man  Months 

"b"  Weights 


1.00 

.09 

.22 

-  .11 

.09 

1.00 

-  .13 

.18 

.22 

-  -13 

1.00 

^=t 

0 

1 

.11 

.18 

-4- 

0 

1 

1.00 

.23 

.21 

.30 

.27 

.27 

.28 

.51 

M 

Multiple  Correlation  Coefficient  with  Adjusted  'V1  Weights 

.49 

20 

Standard.  Error  of  Estimate  with  Adjusted  "b"  Weights 

.68 


2(ke  computation  of  the  adjusted  'V  weights 
Appendix . 


is  shown  in  the  last  page  of  this 
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VARIABLE  9,  LOG1q  TOTAL  COMPUTER  HOURS 
AGAINST  COMPONENTS  OF  UNIQUENESS  INDEX 


Innovation 

Stringent 

Timing 

First 
Effort  on 
Computer 

Prog.  Dev. 
at  More 
Than  1  Loc . 

Intercorrelation  of  Unweighted  Uniqueness 

Components 

12 

Innovation  in  System 

1.00 

.09 

.22 

-  .11 

54 

Stringent  Timing  Requirements 

.09 

1.00 

-  .13 

.18 

64 

First  Programming  Effort  on  Computer 

.22 

-  .13 

1.00 

-  .04 

106 

Program  Developed  at  More  Than  One 
Location 

-  .11 

.18 

-  .04 

1.00 

Table  of  Validity  and  Regression  Coefficients 

Validity  of  Unweighted  Uniqueness  Components 


with  Log^Q  Computer  Hours 

•  35 

•  27 

•  35 

.20 

"b"  Weights 

•45 

.42 

.60 

•  34 

Multiple  Correlation  Coefficient  with  Adjusted  'V1  Weights 

•  56 

21 

Standard  Error  of  Estimate  with  Adjusted  'V1  Weight 


•  TO 


‘The  computation  of  the  adjusted  ,fb"  weights  is  shown  in  the  last  page  of  this 
Appendix . 
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VARIABLE  20,  L0G10  NEW  MACHINE  INSTRUCTIONS 
AGAINST  COMPONENTS  OF  UNIQUENESS 


Innovation 

Stringent 

Timing 

First 
Effort  on 
Computer 

Prog.  Dev, 
at  More 
Than  1  Loc, 

Intoroorrelation  of  Unweighted  Uniqueness 

Components 

12  Innovation  in  System 

1.00 

.09 

.22 

-  .11 

54  Stringent  Timing  Requirements 

.09 

1.00 

-  .13 

.18 

64  First 

.22 

-  .13 

1.00 

-  .04 

106  Program  Developed  at  More  Than  One 
Location 

-  .11 

.18 

-4- 

o 

• 

1 

1.00 

Table  of  Validity  and  Regression  Coefficients 

Validity  of  Unweighted  Uniqueness  Components 
with  Log New  Machine  Instructions 

.16 

.32 

.24 

"b"  Weights 

.43 

ON 

H 

• 

.46 

.39 

Multiple  Correlation  Coefficients  with  Adjusted  "bn  Weight 

.49 

22 

Standard  Error  of  Estimate  with  Adjusted  "b"  Weight 

.64 


22 


The  computation  of  the  adjusted  'V*  weights  is  shown  in  the  last  page  of  this 
Appendix . 
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VARIABLE  121,  LOG10  MONTHS  ELAPSED 
AGAINST  COMPONENTS  OF  UNIQUENESS  INDEX 


Intercorrelation  of  Unweighted  Uniqueness 

Components  Variable  Number 

Innovation 

Stringent 

Timing 

First 
Effort  on 
Computer 

Prog.  Dev. 

at  More 
Than  1  Loc. 

12  Innovation  in  System 

1.00 

.09 

.22 

-  .11 

54  Stringent  Timing  Requirements 

.09 

1.00 

-  .13 

.Id 

64  First  Programming  Effort  on  Computer 
106  Program  Developed  at  More  Than  One 

.22 

-  .13 

1.00 

-  .04 

Location 

-  .11 

.18 

-  .04 

1.00 

Table  of  Validity  and  Regression  Coefficients 

Validity  of  Unweighted  Uniqueness  Components 


with  Log^Q  Total  Man  Months 

•19 

•27 

•  27 

.29 

V  Weights 

.08 

•  17 

.22 

.18 

Multiple  Correlation  Coefficients  with  Adjusted  "b"  Weights 


•50 


23 

Standard  Error  of  Estimate  with  Adjusted  'V1  Weight 

.30 


23, 


The  computation  of  the  adjusted  'Id"  weights  is  shown  in  the  last  page  of  this 
Appendix . 
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VARIABLE  13b,  COSTLINESS  FACTOR 
AGAINST  COMPONENTS  OF  UNIQUENESS  INDEX 


Stringent 

Innovation  Timing 


First  Prog.  Dev. 

Effort  on  at  More 

Computer  Than  1  Loc. 


intercorrelation  of  Unweighted.  Uniqueness 

Components 

12  Innovation  in  System 

52  Stringent  Timing  Requirement 

64  First  Programming  Effort  on  Computer 

lOo  Program  Developed  at  More  Than  One 
Location 


1.00 

.09 

.22 

-  .11 

.09 

1.00 

-  .13 

.18 

.22 

1 

b 

1.00 

O 

1 

.  .11 

-  .18 

-  .04 

1.00 

Table  of  Validity  and  Regression  Coefficients 

Validity  of  Unweighted  Uniqueness  Components 
with  Costliness  Factor 

"b"  Weights 


.29 

.24 

•  30 

.27 

.45 

.40 

.62 

•  52 

Multiple  Correlation  Coefficient  with  Adjusted  'V  Weights 

.52 

24 

Standard  Error  of  Estimate  with  Adjusted  'V1  Weights 

.84 


24The  computation  of  the  adjusted  'V  weights  is  shown  in  the  last  page  of  this 
Appendix . 
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APPENDIX  IX  (Cont'd) 
COMPUTATION  OF  ADJUSTED  WEIGHTS 
FOR  COMPONENTS  OF  THE  UNIQUENESS  INDEX 


After  the  five  major  dependent  variables  were  regressed  against  the  components 
of  the  Uniqueness  Index,  the  resulting  coefficients,  or  "b"  weights,  for  each 
component  were  averaged  and  divided  by  the  smallest  value  to  obtain  the  final 
weights. 


REGRESSION  COEFFICIENTS 


Major  Cost  Variables 

Innovation 

Stringent 

Timing 

First 

Programming 
Effort  on 
Computer 

Program 
Developed  at 
More  Than  1 
Location 

Log  q  Man  Months 

t- 

OJ 

• 

.28 

.51 

.41 

Log^  Computer  Hours 

.45 

.42 

.60 

.34 

Logic  New  Machine 
Instructions 

.43 

.19 

.46 

•  39 

Logic  Months  Elapsed 

.08 

•  17 

.22 

.18 

Costliness  Factor 

.45 

.40 

.62 

•  52 

Sums  of  "b"  Weights 

1.68 

1.46 

2.4l 

1.84 

Component  Weights 

1.2 

1.00 

1.7 

1-3 
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APPENDIX  X 


CORRELATION  AND  REGRESSION  ANALYSIS: 
THE  DEVELOPMENT  OF  ESTIMATION  EQUATIONS 
FOR  THE  MAJOR  DEPENDENT  VARIABLES 
USING  TASK  INDICES  AS  PREDICTORS 


This  Appendix  provides  the  statistical  details  that  underlie  the  estimation 
equations  developed  for  each  of  the  major  dependent  variables,  i.e.,  Logj_o 
Total  Man  Months,  Logqo  Total  Computer  Hours,  Logqo  New  Machine  Instructions, 
Logio  Months  Elapsed,  and  the  Costliness  Factor,  with  the  indices — Uniqueness, 
Job  Difficulty,  Development  Environment  and  Type  of  Job — used  as  a  predictor 
variables . 

The  reader  should  note  that  the  Standard  Errors  of  Estimate  are  in  logarithmic 
form. 
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APPENDIX  X  (Cont'd) 


VARIABLE  1,  LOG1q  TOTAL  MAN  MONTHS 
AGAINST  TASK  INDICES 


Type  of 
Job 


•  15 
.24 
-  .14 
1.00 


.44 

.29 

.27 
•  36 

Multiple  Correlation  Coefficient 

.75 

Standard  Error  of  Estimate 

•52 

Adjusted  Standard  Error  of  Estimate 

•  54 

Prediction  Equation 

**10  \  =  -17X  49  +  .06x150  +  .16X151  +  .27X152  -  1.5V 
Adjusted  Prediction  Equation 

^610  Y1  =  -23Xl49  +  •o8xi50  + *  * **21JC151  +  -36X152  +  1>55 


Job  Development 

Uniqueness  Difficulty  Environment 


Table  of  Intercorrelations 


149  Uniqueness  Index 

1.00 

.28 

+  .25 

150  Job  Difficulty  Index 

.28 

1.00 

+  .28 

151  Development  Environment  Index 

-  .25 

-  .28 

1.00 

152  Type  of  Job  Index 

.15 

.24 

+  .14 

Table  of  Validity  and  Regression  Coefficients 

Validity  Coefficients  with  Log1Q  Man  Months 

.49 

.48 

+  .52 

Beta  Weight 

.30 

.23 

+  -33 

V  Weight 

.17 

.06 

+  .16 

Adjusted  "b"  Weight 

.23 

.08 

+  .21 
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VARIABLE  9,  LOG1q  TOTAL  COMPUTER  HOURS 
AGAINST  TASK  INDICES 


Uniqueness 

Job 

Difficulty 

Development 

Environment 

Type  of 
Job 

Table  of  Intercorrelations 

149  Uni quene  s  s  Index 

1.00 

.28 

+  .25 

.15 

150  Job  Difficulty  Index 

.28 

1.00 

+  .28 

.24 

151  Development  Environment  Index 

-  -25 

-  .28 

1.00 

-  .14 

152  Type  of  Job  Index 

•15 

.24 

+  .14 

1.00 

Table  of  Validity  and  Regression  Coefficients 

Validity  Coefficients  with  Log  Total 
Computer  Hours  1 

•56 

•58 

+  .51 

•  37 

Beta  Weight 

.36 

•35 

+  .29 

•19 

"b"  Weight 

.22 

.10 

+  -15 

•19 

Adjusted  'V  Weight 

.28 

.12 

+  -19 

•  25 

Multiple  Correlation  Coefficient 

•79 

Standard  Error  of  Estimate 

•  52 


Adjusted  Standard  Error  of  Estimate 

•53 


Prediction  Equation 

^lO  Y9  =  *22Xl49  +  ‘10X150 


Adjusted  Prediction  Equation 

Ice10  Y9  =  ,2®X199  +  -12xl50 


+  +  -Wi58  +  ^ 

+  .19X151  +  -25X152  +1.69 
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VARIABLE  20,  LOG  NEW  MACHINE  INSTRUCTIONS 
AGAINST  TASK  INDICES 


Job 

Development 

Type  of 

Uniqueness 

Difficulty 

Environment 

Job 

Table  of  Intercorrelations 

149 

Uniqueness  Index 

1.00 

.28 

+  -25 

•15 

150 

Job  Difficulty  Index 

.28 

1.00 

+  .28 

.24 

151 

Development  Environment  Index 

-  .25 

-  .28 

1.00 

-  .14 

152 

Type  of  Job  Index 

.15 

.24 

+  .14 

1.00 

Table  of  Validity  and  Regression  Coefficients 


Validity  Coefficients  with  Log1Q  New 
Instructions 

•47 

+  .40 

•39 

.52 

Beta  Weight 

•34 

.26 

+  .25 

.26 

,fb"  Weight 

•19 

.06 

+  .11 

.23 

Adjusted  ,lb"  Weights 

.29 

.09 

+  -13 

.32 

Multiple  Correlation  Coefficient 

•70 

Standard  Error  of  Estimate 

.54 

Adjusted  Standard  Error  of  Estimate 

•57 


Prediction  Equation 

“*10  1 20  •  -19J!l49  *  •'**150  +  -ml51  *  -23X152  *  3M 


Adjusted  Prediction  Equation 

L°S10  Y20  =  -29Xi49  +  -°9X150  +  .13X151  +  .32X^2  +  3-31 
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VARIABLE  121,  LOG^Q  MONTHS  ELAPSED 
AGAINST  TASK  INDICES 


Job  Development  Type  of 


Uniqueness 

Difficulty 

Environment 

Job 

Table  of  Inter correlations 

149  Uniqueness  Index 

1.00 

.28 

+  .25 

.15 

150  Job  Difficulty  Index 

.28 

1.00 

+  .28 

.24 

151  Development  Environment  Index 

-  .25 

-  .28 

1.00 

-  .14 

152  Type  of  Job  Index 

•15 

.24 

+  .14 

1.00 

Table  of  Validity  and  Regression 

Coefficients 

Validity  Coefficients  with  Log 
Elapsed 

Months 

.48 

•55 

+  .38 

.36 

Beta  Weight 

•  31 

•36 

+  .18 

.21 

V  Weight 

.08 

.04 

+  .04 

.08 

Adjusted  "b"  Weight 

.11 

.06 

+  .05 

.12 

Multiple  Correlation  Coefficient 

•  70 

Standard  Error  of  Estimate 

.25 

Adjusted  Standard  Error  of  Estimate 

.26 

Prediction  Equation 

^g10  YI21  =  *°®xi49  +  #0^X150  +  ,0^X151  +  #0®X152  + 
Adjusted  Prediction  Equation 

**10  YI21  =  -mi49  +  -06X150  +  *05X151  +  •32X152  +  ^ 
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APPENDIX  X  (Cont'd) 
VARIABLE  151*,  COSTLINESS  FACTOR 
AGAINST  TASK  INDICES 


Uniqueness 

Job 

Difficulty 

Development 

Environment 

Type  of 
Job 

Table  of  Intercorrelations 

149 

Uniqueness  Index 

1.00 

.28 

-  .25 

•15 

150 

Job  Difficulty  Index 

.28 

1.00 

-  .28 

.24 

151 

Development  Environment  Index 

-  .25 

-  .28 

1.00 

-  .14 

152 

Type  of  Job  Index 

.15 

.24 

-  .14 

1.00 

Table  of  Validity  and  Regression  Coefficients 


Validity  Coefficients  with  Costliness 


Factor 

•52 

•59 

-  .53 

•  45 

Beta  Weight 

•  30 

•  35 

-  -32 

.28 

V  Weight 

5-83 

2.90 

-3.81 

8-39 

Adjusted  "b"  Weight 

18.93 

3.58 

-5.89 

10.37 

Multiple  Correlation  Coefficient 

.82 


Standard  Error  of  Estimate 

12.0 


Adjusted  Standard  Error  of  Estimate 

12.7 


Prediction  Equation 

1l9l  -  5.63^  *  2.90^  -  3.81^  ♦  S.39^52  -  88.1.7 

Adjusted  Prediction  Equation 

Y154  =  18.93Xi49  +  3.58x150  +  5.89X151  +  10.37X152  +  66.96 
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APPENDIX.  XI 


THE  COMPUTATIONAL  PROCEDURES  TO  CORRECT 
FOR  CURVILINEARITY  IN  THE  STANINE  BANDS 


The  Stanine  bands  that  are  shown  in  Figures  14,  15,  16,  17,  and  18,  Section 
XVII,  are  straight-line  approximations  based  on  constant  multiples  of  the 
Standard  Error  of  Estimate  (o  ),  e.g.,  the  limits  of  the  "5"  band  are  +  O.25o 
about  the  prediction  line .  * 

In  fact,  the  Standard  Error  of  Estimate  is  not  constant  but  expands  as  the 
predictor  indices  differ  from  their  respective  means.  The  result  is  called 
the  Standard  Error  of  Prediction  (or,  sometimes,  the  Standard  Error  of  Fore¬ 
cast)  .  Since  the  corrections  are  unique  for  every  combination  of  predictor 
values,  the  calculations  are  rather  complex(l4). 

The  reader  should  carefully  note,  however,  that  both  the  dependent  variables 
and  the  Standard  Errors  in  the  five  estimation  equations  are  in  logarithmic 
form.  This  means  that,  when  the  Stanine  band  limits  are  computed  (by  adding 
and  subtracting  an  appropriate  multiple  of  the  Standard  Error  from  the  equation 
value),  it  is  necessary  to  add  and  subtract  these  values  retained  in  their 
logarithmic  form  before  converting  to  raw  values. 

The  combination  of  these  procedures  produces  corrections  that  are  best  illus¬ 
trated  with  a  pair  of  examples  in  the  following  table.  These  have  been 
worked  out  for  two  cases  of  the  dependent  variable  Total  Man  Months,  one  at 
33  man  months,  and  the  other  at  5^-0  man  months.  In  the  first  example,  all  the 
predictor  indices  are  near  their  respective  means. 

Several  aspects  of  the  computed  corrections  are  worthy  of  note: 

a.  The  percentage  corrections  are  quite  small  when  the  predictor 
indices  are  near  their  respective  means,  but  enlarge  as  the 
predictors  deviate  from  their  means .  The  corrections  are  added 
to  the  computed  estimate  above  the  prediction  line  and  subtracted 
below  it. 

b.  The  use  of  logarithmically  transformed  variables  also  results  in 
an  increased  percentage  correction  the  farther  a  particular  band 
is  from  the  prediction  line,  e.g.,  a  larger  correction  at  the  "7" 
band  than  at  the  "5"  band.  These  errors  are  also  asymmetrical 

in  raw  values,  although  the  percentage  corrections  are  symmetrical. 

(see  the  last  two  columns  in  the  charts  for  each  example.) 
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APPENDIX  XI  (Cont'd) 


Although  these  two  examples  do  not  demonstrate  the  point,  the  percentage 
corrections  (see  the  5^0 -man- month  case  in  the  chart)  level  off  i.e.,  reach 
an  asymptote  for  larger  values,  while  the  raw  errors  continue  to  increase 
proportionately  (The  same  is  true  for  cases  below  33  man  months,  although 
we  have  not  worked  out  an  example) . 

To  allow  the  reader  to  determine  the  corrections  for  an  individual  estima¬ 
tion  and  Stanine  band,  a  work  sheet  has  been  provided.  In  addition  to  the 
computed  value  of  each  index,  the  Standard  Error  of  Estimate  is  required  for 
each  equation,  and  has  been  abstracted  from  Appendix  X. 

Dependent  Variable  Standard  Error  of  Estimate  (in  Log^g) 

Total  Man  Months  0.54 

Total  Computer  Hours  0.53 

New  Machine  Instructions  0.57 

Months  Elapsed  0 .26 

Costliness  Factor  0*59 

To  obtain  the  corrected  limits  of  the  Stanine  bands  for  an  individual  case, 
the  Standard  Error  of  Prediction  should  be  multiplied  by  the  following  values 


Stanine  Band 


Multiple  of  the  Standard 

Error  of  Prediction 


9  (upper  limit) 
8  (upper  limit 
7  (upper  limit) 
6  (upper  limit 
5  (upper  limit) 
5  (lower  limit) 
4  (lower  limit) 
3  (lower  limit) 
2  (lower  limit) 
1  (lower  limit) 


Open-Ended 


Open-Ended 


+1.75 
+1 .25 
+0.75 
+0.25 
-0.25 
-0.75 
-1.25 
-1.75 
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13 

9 

5 

2 

-2 
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-9 
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c 

HI  4  (\J  H  O 

<§•  S 

Stanine 

Bands 

9  (upper  limit) 

8  (upper  limit) 

7  (upper  limit) 

6  (upper  limit) 

5  (upper  limit) 

5  (lower  limit) 

4  (lower  limit) 

3  (lower  limit) 

2  (lower  limit) 

1  (lower  limit) 

Correction  as 
a  Percent  of 
Straight-line 
Approximation 

IA  O  h  C\1  C\l  t—  O  UA 

HHOOOOHH 

l  i  i  1 
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oh. 
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APPENDIX  XI  (Cont'd) 


A  WORKSHEET  FOR  COMPUTING  THE  STANDARD  ERROR  OF  PREDICTION 
FROM  THE  STANDARD  ERROR  OF  ESTIMATE  WHEN  USING  TASK  INDICES  AS  PREDICTORS 


Indices 

Va.lues  of  Predictor 
Indices  (Xj ) 

Means  of  Predictor 
Indices  (X^) 

Deviation  of  Predictors_from 
their  Means  X.  =  (X^  -  X^) 

Uniqueness  Index 

2.09189 

Job  Difficulty  Index 

5.9868^ 

Development  Environment  Index 

-4.74220 

Job  Type  Index 

.04493 

1 

1 _ 

Coefficients 

of  Multiplication  Result  of  Multiplying 


(*1  -  = 

H* 

u 

_ 1 

.00887 

n 

1 

£ 

(Xg  -  Xg)  = 

.00180 

(x3  -  x3)  = 

(x3  -  x3)  = 

.00578 

(x4  -  %)  = 

(xu  -  xk)  = 

.02196 

(xi  -  xx)  = 

(Xg  -  Xg)  = 

-.OOI65 

(xX  -  X,)  = 

(x3  -  x3)  = 

-.00250 

(X1  -  Xj.)  = 

(x4  -  x^)  = 

-.00209 

(Xg  -  x2)  = 

(x3  -  x3)  = 

-.00138 

(Xg  -  Xg)  = 

(x4  -  *4)  = 

-.00236 

(x3  -  x3)  = 

(x4  -  *4)  = 

-.00150 

Sura  of  (Xj)  (Xj^)  (dJk) 
Add  Constant 
T 

V T 

Enter  Standard  Error  of  Estimate  (Log^) 
Multiply  by  n/t  to  obtain  Standard  Error  of  Prediction  (Log^) 


1.01351 
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XII 


THE  SENSITIVITY  OF  THREE  DEPENDENT  VARIABLES  TO  CHANGES 
IN  THE  TASK  INDICES:  A  SUMMARY 


This  Appendix  summarizes  the  sensitivity  of  the  dependent  variables — Total 
Man  Months,  Months  Elapsed,  and  Total  Computer  Hours — to  changes  in  the 
predictor  indices.  Briefly,  the  computational  procedure  involves  a  determi¬ 
nation  of  the  equation  estimate  for  the  dependent  variable  with  the  four 
predictor  indices  equal  to  their  respective  means,  and,  then,  varying  each 
index  by  +10  percent,  +1  standard  deviation,  -10  percent,  and  -1  standard 
deviation,  while  holding  the  remaining  3  predictor  indices  constant. 
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SUMMARY  OF  SENSITIVITY  RELATIONSHIPS 
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Hours  Development  Environment  +76  i  18  -15 

(Variable  9)  Type  of  Job  -30  j  negligible  negligible 
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13  ABSTRACT 

The  report  embodies  the  latest  results  of  a  continuing  research  effort  directed 
toward  the  development  of  management  guidelines,  standards,  and  techniques  in  the 
field  of  computer  programming.  The  work  is  based  upon  earlier  studies  at  System 
Development  Corporation  that  included:  the  definition  of  variables  affecting 
computer  programming  costs;  the  design  of  a  questionnaire  as  an  aid  to  collecting 
data  on  completed  jobs;  and  the  exploratory  statistical  analysis  of  27  completed 
computer  programming  jobs  to  develop  preliminary  cost- estimation  relationships. 

The  present  report  is  focused  upon  the  statistical  analysis  of  'jk  completed 
computer  programming  jobs  in  terms  of  their  resource-costs  and  related  variables, 
e.g. ,  man  months,  computer  hours.  The  primary  results  developed  in  this  analysis 
ere:  indices  of  job  difficulty,  job  type,  development  environment,  and  job 
uniqueness;  a  "costliness"  factor  that  permits  programming  tasks  to  be  ranked  in 
this  respect;  weighted  composites  of  the  indices  for  estimating  the  cost  of 
particular  programming  jobs;  and  scoring  and  confidence-band  techniques  for 
blending  intuitive  managerial  judgments  with  the  formal  cost- estimation  procedures. 
Supplementary  findings  include  indications  of  the  relative  sensitivity  of  job 
cost  to  changes  in  the  values  for  the  indices,  and  preliminary  comparisons  of 
resource  usage  between  programs  produced  in  machine- oriented  or  procedure-oriented 
languages.  Also,  recommendations  are  made  for  future  research,  including:  the 
collection  of  more  accurate  and  current  data  on  programming  jobs  during  the 
production  cycle,  and  the  development  of  a  census  of  computer  programming,  to 
enable  the  design  of  precise  sampling  experiments  for  subsequent  analyses,  (author) 
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